Photo realistic rendering of smile image after treatment

ABSTRACT

A method may include: receiving facial image of the patient that depicts the patient&#39;s teeth; receiving a 3D model of the patient&#39;s teeth; determining color palette of the depiction of the patient&#39;s teeth; coding 3D model of the patient&#39;s teeth based on attributes of the 3D model; providing the 3D model, the color palette, and the coded 3D model to a neural network; processing the 3D model, the color palette, and the coded 3D model by the neural network to generate a processed image of the patient&#39;s teeth; simulating specular highlights on the processed image of the patient&#39;s teeth; and inserting the processed image of the patient&#39;s teeth into a mouth opening of the facial image.

This application claims the benefit of U.S. Provisional Application No.62/674,524, filed May 21, 2018, which application is incorporated hereinby reference.

BACKGROUND

In the design of virtual representations of living beings, an “uncannyvalley” can relate to the extent a virtual object's resemblance to aliving being corresponds to emotional responses to the virtual object.The concept of an uncanny valley may suggest humanoid virtual objectswhich appear almost, but not exactly, like real human beings (robots, 3Dcomputer animations, lifelike dolls, etc.) elicit uncanny, or strangelyfamiliar, feelings of eeriness and revulsion in observers. A virtualobject that appears “almost” human risks may elicit cold, eerie, and/orother non-emotional feelings in viewers.

In the context of treatment planning, the uncanny valley problem maycause people to negatively react to humanoid representations ofthemselves. As an example, people viewing a 3D virtual representation ofthemselves after an orthodontic treatment plan may be confronted with anunfamiliar, robotic, or non-humanoid view of themselves.

These issues may undermine perceptions of treatment planning proposalsand/or lead to negative perceptions of treatment planning proposals.These issues may influence decision making with respect to the properalignment of teeth in a human patient when three-dimensional (3D)renderings of digital models of patient's teeth are combined withtwo-dimensional (2D) images or photos of a patient.

Furthermore, orthodontic treatment planning processes may not take intoaccount facial relationships between the positions and orientations ofteeth and the shape and position of facial features of a patient.

Systems and methods that reduce the uncanny valley reaction and takeinto account the relationships between the positions and orientations ofteeth and the shape and position of facial features of a patient couldhelp in increasing the effectiveness and acceptance of orthodontictreatment, particularly orthodontic treatments involving virtualrepresentations and/or virtual 3D models of living beings before, duringand/or after the application of orthodontic treatment plans.

SUMMARY

This disclosure generally relates to system and methods of correctingmalocclusions of teeth. More particularly, the disclosure relates tosystem and methods of accurately and realistically depicting 3D bitemodels of teeth in a 2D image of patient and systems and methods ofdetermining the final orthodontic and restorative object positions forteeth.

Systems and methods are described herein to more closely integrate 3Dbite models into 2D images of a patient to aid in reducing oreliminating the uncanny valley reaction and aid in providing betterdecision making with respect to the proper alignment of teeth.

In addition, orthodontic systems and methods are introduced thatevaluate the shape of a patient's face and the relationships between thepositions and orientations of teeth and the shape and position of facialfeatures of a patient in the treatment planning processes such that thefinal orthodontic and resorted position and shape of the patient teethmore closely match an ideal position with respect to each particularpatient.

A method of orthodontically treating a patient is disclosed. The methodmay include building a 3D model of the patient's dentition and formingan image a patient including at least a portion of the patient's faceand including at least a portion of the patient's dentition. The methodmay also include selecting a first set of reference points on the 3Dmodel of the patient's dentition. The method may include selecting asecond set of reference points on the dentition of the image of thepatient and combining the 3D model of the patient's dentition with theimage of the patient. The method may also include aligning the first setof reference points on the 3D model of the patient's dentition with thesecond set of reference points on the image of the patient.

Computer-readable media, computer systems having processors and memory,and computer-implemented methods of orthodontically treating a patientmay comprise: building a three-dimensional model of the patient'sdentition; forming an image of the patient including at least a portionof the patient's face and including at least a portion of the patient'sdentition; selecting a first set of reference points on thethree-dimensional model of the patient's dentition; selecting a secondset of reference points on the dentition of the image of the patient;combining the three-dimensional model of the patient's dentition withthe image of the patient; and aligning the first set of reference pointson the three-dimensional model of the patient's dentition with thesecond set of reference points on the image of the patient.

In some implementations, at least one reference point in the first setof reference points corresponds to a respective reference point in thesecond set of reference points. In some implementations, thecorresponding reference points are at similar locations on thethree-dimensional model and the image of the patient. In someimplementations, the similar locations are the gingival apex of at leasttwo teeth. In some implementations, the similar locations are themidpoint of incisal edges. In some implementations, the similarlocations are cusp tips.

In some implementations, the image of the patient is a two-dimensionalimage of the patient. In some implementations, contours of the teeth inthe three-dimensional model of the patient's dentition are aligned withcontours of teeth in the image of the patient. In some implementations,the aligning includes minimizing the square of the distance between thefirst set of reference points on the three-dimensional model and thesecond set of reference points on the image of the patient.

Computer-readable media, computer systems having processors and memory,and computer-implemented methods of orthodontically treating a patientmay comprise: combining a three-dimensional bite model with an image ofa patient; rendering the three-dimensional bite model according to afirst set of parameters; determining a quality metric for the renderedthree-dimensional bite model as compared to the image of the patient;selecting a second set of parameters to improve the quality metric forthe rendered three-dimensional bite model as compared to the image ofthe patient; and rendering the three-dimensional bite model according toa second set of parameters.

In some implementations, the method may further comprise: determining aquality metric for the rendered three-dimensional bite model as comparedto the image of the patient; and comparing the quality metric to athreshold value; determining if the quality metric at least meets thethreshold value; and generating a final composite image of the patientbased on the three-dimensional bite model rendered according to a secondset of parameters and the image of the patient, if the quality metric atleast meets the threshold value.

In some implementations, the image of the patient is a two-dimensionalimage. In some implementations, rendering the three-dimensional bitemodel according to the second set of parameters includes rendering thethree-dimensional bite model according to an interim position of theteeth according to a treatment plan for moving the patient's teeth froman initial position towards a final position.

In some implementations, rendering the three-dimensional bite modelaccording to the second set of parameters includes rendering thethree-dimensional bite model according to a final position of the teethaccording to a treatment plan for moving the patient's teeth from aninitial position towards a final position.

In some implementations, the first and second parameters include a colorintensity. In some implementations, the first and second parametersinclude a luminance intensity. In some implementations, the first andsecond parameters include a tooth whiteness. In some implementations,the first and second parameters include a blur. In some implementations,the first and second parameters include a shadow filter.

Computer-readable media, computer systems having processors and memory,and computer-implemented methods for correcting malocclusions of apatient's teeth comprising: receiving an initial position of a patientteeth; determining an interim final orthodontic position of a patient'steeth; applying at least one interim restorative object to at least oneof the patient's teeth; determining an interim tooth mass loss for theat least one of the patient's teeth; determining a final orthodonticposition of a patient's teeth; applying at least one final restorativeobject to at least one of the patient's teeth; and determining a finaltooth mass loss for the at least one of the patient's teeth, wherein thefinal tooth mass loss is less than the interim tooth mass loss.

In some implementations determining an interim tooth mass loss for theat least one of the patient's teeth includes: determining an initialmass of the patient's tooth before preparing the tooth for the interimrestorative object; determining a prepared mass of the patient's toothbased on the shape of the tooth when prepared for receiving the interimrestorative object; and subtracting the prepared mass from the initialmass.

In some implementations, determining tooth mass loss includesdetermining the volume loss of the tooth. In some implementations,determining the volume loss of the tooth for the at least one of thepatient's teeth includes: determining an initial volume of the patient'stooth before preparing the tooth for the interim restorative object;determining a prepared volume of the patient's tooth after preparing thetooth for the interim restorative object; and subtracting the preparedvolume from the initial mass.

In some implementations, determining tooth mass loss includesdetermining the tooth mass loss of the crown of the tooth. In someimplementations, the interim restorative object is a crown. In someimplementations, the final restorative object is a veneer.

Computer-readable media, computer systems having processors and memory,and computer-implemented methods of determining a final position of apatient's teeth comprising: receiving an initial position of a patientteeth; determining an interim final orthodontic position of thepatient's teeth; receiving an image of the patient's face; selecting afirst set of reference objects on the image of the patient's face;selecting a second set of reference objects on the patient's teeth; andrevising the interim final orthodontic position of the patient's teethbased on distances between points in the first set of reference objectsand the second set of reference objects.

In some implementations, the first set of reference objects include afacial midline and the second set of reference objects include a dentalmidline. In some implementations, revising the interim final orthodonticposition of the patient's teeth includes changing the position of thepatient's teeth in the interim final orthodontic position such that adistance between the facial midline and the dental midlines is less thana threshold value.

In some implementations, the first set of reference objects include alocation of the inferior boarder of the upper lip at the facial midlineand the second set of reference objects include an incisal edgeposition. In some implementations, revising the interim finalorthodontic position of the patient's teeth includes changing theposition of the patient's teeth in the interim final orthodonticposition such that a distance between the location of the inferiorboarder of the upper lip at the facial midline and the incisal edgeposition is less than a threshold value.

In some implementations, the first set of reference objects include alocation of the superior boarder of the lower lip at the facial midlineand the second set of reference objects include an incisal edgeposition. In some implementations, revising the interim finalorthodontic position of the patient's teeth includes changing theposition of the patient's teeth in the interim final orthodonticposition such that a distance between the location of the superiorboarder of the lower lip at the facial midline and the incisal edgeposition is less than a threshold value.

In some implementations, the first set of reference objects include alocation of the inferior boarder of the upper lip at the facial midlineand the second set of reference objects include a gingival zenith of acentral incisor. In some implementations, revising the interim finalorthodontic position of the patient's teeth includes changing theposition of the patient's teeth in the interim final orthodonticposition such that a distance between the location of the inferiorboarder of the upper lip at the facial midline and the gingival zenithof the central incisor is less than a threshold value.

In some implementations, widths of the central incisors, lateralincisors, and canines in the interim final position are modified basedon facial type. In some implementations, the modification includesapplication of restorative objects.

In some implementations, widths of the central incisors, lateralincisors, and canines in the interim final position are modified basedon an inter-canine width and facial type.

In some implementations, facial type is determined based on a distancebetween the patient's glabella and chin and the distance between thepatient's right and left cheekbone prominences.

Computer-readable media, computer systems having processors and memory,and computer-implemented methods of orthodontically treating a patient'steeth comprising: receiving facial image of the patient that depicts thepatient's teeth; receiving a 3D model of the patient's teeth;determining color palette of the depiction of the patient's teeth; colorcoding 3D model of the patient's teeth based on attributes of the 3Dmodel; providing the 3D model, the color palette, and the color-coded 3Dmodel to a neural network; processing the 3D model, the color palette,and the color-coded 3D model by the neural network to generate aprocessed image of the patient's teeth; and inserting the processedimage of the patient's teeth into a mouth opening of the facial image.

In some implementations, a spline may be formed at the edge of the innerlips to define the mouth opening of the facial image. In someimplementations, the neural network is trained using facial images ofpeople that depict their teeth. In some implementations, the processedimage of the patient's teeth is blurred. In some implementations, theblurring occurs after inserting the processed image of the patient'steeth into the mouth opening of the facial image. In someimplementations, the blurring is alpha channel blurring.

In some implementations, generating the color palette comprises:blurring the depiction of the patient's teeth from the facial image.

In some implementations, the blurring is a Gaussian blur.

In some implementations, the facial image is a 2D facial image. In someimplementations, color coding the 3D model comprises coding a colorchannel of a plurality of pixels of a 2D rendering of the 3D model withattributes of the 3D model or the facial image. In some implementations,the attributes are one or more of the brightness of the patient's teethat each pixel location, the angle of the surface of the 3D model withrespect to the facial plane at each pixel location, and the dentalstructure type of the 3D model at each pixel location. In someimplementations, the brightness of the patient's teeth location at eachpixel location is determined based on the brightness of a blurreddepiction of the patient's teeth from the facial image. In someimplementations, the dental structure is one or more of an identity ofeach tooth or the gingiva in the 3D model at each pixel location

Computer-readable media, computer systems having processors and memory,and computer-implemented methods of orthodontically treating a patient'steeth comprising: receiving an image of a mouth region of a patient'sface; extracting teeth contours within the image of the mouth region ofthe patient's face; locating a mouth opening within the image of themouth region of the patient's face; extracting the tooth contours from a3D model of the patient's teeth; and aligning the tooth contours fromthe 3D model with the tooth contours of the teeth within the image ofthe mouth region of the patient's face.

In some implementations, a rendering of the 3D model may be insertedinto the mouth opening based on the alignment of the tooth contours fromthe 3D model with the tooth contours of the teeth within the image ofthe mouth region of the patient's face.

In some implementations, receiving the image of the mouth region of thepatient's face comprises: receiving a facial image of the patient;identifying facial landmarks on the facial image, the facial landmarksincluding lip landmarks and other landmarks; and cropping the facialimage around the lip landmarks to exclude the other landmarks.

In some implementations, the other landmarks include one or more of eye,nose and facial outline landmarks.

In some implementations, extracting the teeth contours within the imageof the mouth region of the patient's face comprises: detecting the toothcontours using a convolutional neural network.

In some implementations, the convolutional neural network comprises aholistic edge detection deep learning model.

In some implementations, each pixel of the tooth contours has a value.In some implementations, the tooth contours are binarized. In someimplementations, binarizing the tooth contours comprises: comparing thevalue of each pixel to a threshold; and assigning a new value to eachpixel, the new value being a first value if the pixel is greater thanthe threshold and a second value of the value is less than thethreshold.

In some implementations, the tooth contours are thinned. In someimplementations, thinning the tooth contours comprises reducing thewidth of the tooth contours to a single pixel at each location along thetooth contours.

In some implementations, aligning the tooth contours from the 3D modelwith the tooth contours of the teeth within the image of the mouthregion of the patient's face comprises: using anexpectation-maximization algorithm to align the tooth contour from the3D model with the tooth contours of the teeth within the image, whereduring an expectation-step, each pixel on the tooth contours from the 3Dmodel is matched to a similar pixel on the contours of the 3D toothmodel and during a maximization step, the teeth are adjusted in one ormore of translation and rotation in one or more of three orthogonaldirection to minimize the total discrepancies between pixels of thetooth contours from the 3D model and the tooth contours of the teethwithin the image.

In some implementations, locating the mouth opening within the image ofthe mouth region of the patient's face comprises: detecting the toothcontours using a convolutional neural network; binarizing the toothcontours; and thinning the tooth contours.

In some implementations, the convolutional neural network comprises aholistic edge detection deep learning model.

In some implementations, each pixel of the tooth contours has a value.

In some implementations, binarizing the tooth contours comprises:comparing the value of each pixel to a threshold; and assigning a newvalue to each pixel, the new value being a first value if the pixel isgreater than the threshold and a second value of the value is less thanthe threshold.

In some implementations, thinning the tooth contours comprises reducingthe width of the tooth contours to a single pixel at each location alongthe tooth contours.

In some implementations, methods further include: forming a firstplurality of connected splines along a lower lip portion of the lipcontour, the plurality of connected splines starting at a first end ofthe lip contour and ending at a second end of the lip contour; andforming a second plurality of connected splines along an upper lipportion of the lip contour, the plurality of connected splines startingat the first end of the lip contour and ending at the second end of thelip contour, wherein the first and second lip splines define the mouthopening.

Computer-readable media, computer systems having processors and memory,and computer-implemented methods of orthodontically treating a patient'steeth, comprise: receiving a facial image a patient; identifying faciallandmarks on the facial image; generating a facial midline based on thelandmarks; forming a facial midline plane based on the facial midline;receiving a 3D tooth model having a dental midline; and aligning the 3Dtooth model with the facial image of the patient by aligning the dentalmidline with the facial midline plane.

In some implementations, the landmarks are symmetric landmarks. In someimplementations, each of the symmetric landmarks is a midpoint between acorresponding pair of facial landmarks, a first of the pair identifyinga feature on a left side of the patient's face and a second of the pairidentifying the same feature on the right side of the face.

In some implementations, the landmarks are central landmarks.

In some implementations, each of the central landmarks is one of a nasalridge landmark; a nose tip landmark, a center lip landmark, a centerchin landmark. In some implementations, the landmarks are symmetriclandmarks and central landmarks.

In some implementations, generating a facial midline based on thelandmarks comprises: generating a plurality of interim facial midlinesand determining the R-squared fit between each of the facial midlinesand the facial landmarks; and wherein the facial midline is the interimfacial midline with the highest R-squared fit.

In some implementations, generating a facial midline based on thelandmarks comprises: generating a plurality of interim facial midlinesand determining a sum of the square of the distances of the facialmidlines and the facial landmarks; and wherein the facial midline is theinterim facial midline with the lowest sum of the square of thedistances.

In some implementations, one or more of the facial landmarks is assigneda weigh used in determining a sum of the square of the distances of thefacial midlines and the facial landmarks.

In some implementations, aligning the 3D tooth model with the facialimage of the patient by aligning the dental midline with the facialmidline plane comprise one or more of rotation the facial image,rotating the 3D model, and translating the 3D model.

Computer-readable media, computer systems having processors and memory,and computer-implemented methods may include: gathering athree-dimensional model of the patient's dentition, thethree-dimensional model comprising a virtual representation of thepatient's dentition at a specific treatment stage of an orthodontictreatment plan; gathering an image of the patient, the image includingat least a portion of the patient's face and including at least aportion of the patient's dentition; receiving first identifiers of afirst set of reference points modeled on the three-dimensional model ofthe patient's dentition, the first set of reference points correspondingto a set of anatomical points on the patient's dentition; receivingsecond identifiers of a second set of reference points represented onthe dentition of the image of the patient, the second set of referencepoints corresponding to the set of anatomical points on the patient'sdentition; projecting the image of the patient's dentition into athree-dimensional space to create a projected 3D model of the image ofthe patient's dentition; aligning the first set of reference points onthe three-dimensional model of the patient's dentition with the secondset of reference points on projected model of the image of the patient;providing instructions to display a modified image of the patient, themodified image representing thereon the aligned first and second sets ofreference points.

In some implementations, at least one reference point in the first setof reference points corresponds to a respective reference point in thesecond set of reference points. In some implementations, the anatomicalpoints are at similar locations on the three-dimensional model and theimage of the patient. In some implementations, the similar locations arethe gingival apex of at least two teeth. In some implementations, thesimilar locations are the midpoint of incisal edges. In someimplementations, wherein the similar locations are cusp tips.

In some implementations, the image of the patient is a two-dimensionalimage of the patient. In some implementations, contours of the teeth inthe three-dimensional model of the patient's dentition are aligned withcontours of teeth in the image of the patient. In some implementations,the aligning includes minimizing the square of the distance between thefirst set of reference points on the three-dimensional model and thesecond set of reference points on the projected model of the image ofthe patient.

In some implementations, the specific treatment stage is an estimatedintermediate stage or an estimated final stage of the orthodontictreatment plan. In some implementations, image represents the patients'dentition before the orthodontic treatment plan. In someimplementations, the image is captured from a scanner or uploaded from acomputer or a mobile phone. In some implementations, the image isuploaded over a network connection.

Computer-readable media, computer systems having processors and memory,and computer-implemented methods may include: identifying, using a firstthree-dimensional (3D) representation of a patient's teeth, an initialposition of the patient teeth; determining, using a virtualrepresentation of one or more force systems applied to the patient'steeth, an estimated interim final orthodontic position of the patient'steeth; gathering one or more images of the patient's face; identifyingone or more facial reference objects on the image of the patient's face,the facial reference objects corresponding to a physical or anatomicalfeature providing a first reference position to the patient's face;identifying one or more dental reference objects on the first 3Drepresentation of the patient's teeth, the one or more dental referenceobjects corresponding to physical or anatomical feature providing asecond reference position to the patient's dentition; identifying arelationship between the one or more facial reference objects and theone or more dental reference objects; modifying the estimated interimfinal orthodontic position of the patient's teeth in the first 3Drepresentation based on the relationship between the one or more facialreference objects and the one or more dental reference objects; andproviding instructions to integrate a modified estimated interim finalorthodontic position based on the relationship between the one or morefacial reference objects and the one or more dental reference objects.

In some implementations, the one or more facial reference objectsinclude a facial midline and the one or more dental reference objectsinclude a dental midline. In some implementations, modifying theestimated interim final orthodontic position of the patient's teethincludes changing the position of the patient's teeth in the estimatedinterim final orthodontic position such that a distance between thefacial midline and the dental midlines meets or does not exceed athreshold value. In some implementations, the one or more facialreference objects include a location of the inferior boarder of theupper lip at the facial midline and the one or more dental referenceobjects include an incisal edge position. In some implementations,modifying the estimated interim final orthodontic position of thepatient's teeth includes changing the position of the patient's teeth inthe estimated interim final orthodontic position such that a distancebetween the location of the inferior boarder of the upper lip at thefacial midline and the incisal edge position meets or does not exceed athreshold value.

In some implementations, the one or more facial reference objectsinclude a location of the superior boarder of the lower lip at thefacial midline and the one or more dental reference objects include anincisal edge position.

In some implementations, modifying the estimated interim finalorthodontic position of the patient's teeth includes changing theposition of the patient's teeth in the estimated interim finalorthodontic position such that a distance between the location of thesuperior boarder of the lower lip at the facial midline and the incisaledge position meets or does not exceed a threshold value. In someimplementations, the one or more facial reference objects include alocation of the inferior boarder of the upper lip at the facial midlineand the one or more dental reference objects include a gingival zenithof a central incisor. In some implementations, modifying the estimatedinterim final orthodontic position of the patient's teeth includeschanging the position of the patient's teeth in the interim finalorthodontic position such that a distance between the location of theinferior boarder of the upper lip at the facial midline and the gingivalzenith of the central incisor meets or does not exceed a thresholdvalue. In some implementations, widths of the central incisors, lateralincisors, and canines in the interim final position are modified basedon facial type. In some implementations, the modification includesapplication of restorative objects.

In some implementations, widths of the central incisors, lateralincisors, and canines in the interim final position are modified basedon an inter-canine width and facial type. In some implementations,facial type is determined based on a distance between the patient'sglabella and chin and the distance between the patient's right and leftcheekbone prominences.

In some implementations, instructions to design or manufacture anorthodontic appliance using the modified estimated interim finalposition are provided.

INCORPORATION BY REFERENCE

All publications, patents, and patent applications mentioned in thisspecification are herein incorporated by reference to the same extent asif each individual publication, patent, or patent application wasspecifically and individually indicated to be incorporated by reference.

BRIEF DESCRIPTION OF THE DRAWINGS

The novel features of the invention are set forth with particularity inthe appended claims. A better understanding of the features andadvantages of the present invention will be obtained by reference to thefollowing detailed description that sets forth illustrative embodiments,in which the principles of the invention are utilized, and theaccompanying drawings of which:

FIG. 1A illustrates a tooth repositioning appliance, in accordance withone or more embodiments herein;

FIG. 1B illustrates a tooth repositioning system, in accordance with oneor more embodiments herein;

FIG. 1C illustrates a method of orthodontic treatment using a pluralityof appliances, in accordance with one or more embodiments herein;

FIG. 2 illustrates a method for designing an orthodontic appliance, inaccordance with one or more embodiments herein;

FIG. 3 illustrates a method for planning an orthodontic treatment, inaccordance with one or more embodiments herein;

FIG. 4 is a simplified block diagram of a system for designing anorthodontic appliance and planning an orthodontic treatment, inaccordance with one or more embodiments herein;

FIG. 5A depicts a method of building a composite image, in accordancewith one or more embodiments herein;

FIG. 5B depicts a method of building a composite image, in accordancewith one or more embodiments herein;

FIG. 6 depicts a two-dimensional (2D or 2-D) image of a patient, inaccordance with one or more embodiments herein;

FIG. 7 depicts a three-dimensional (3D or 3-D) bite model of a patient'steeth, in accordance with one or more embodiments herein;

FIG. 8 depicts the selection of reference points on the 2D image of apatent, in accordance with one or more embodiments herein;

FIG. 9 depicts the selection of reference points on the 3D bite model ofthe patient's teeth, in accordance with one or more embodiments herein;

FIG. 10 depicts the integration of the 3D bite model into the 2D imageof the patient to create a composite image, in accordance with one ormore embodiments herein;

FIG. 11 depicts a close up of a composite image of the 3D bite model andthe 2D image of the patient with overlaid contours, in accordance withone or more embodiments herein;

FIG. 12 depicts a composite image of the 3D bite model in a finalposition and the 2D image of the patient, in accordance with one or moreembodiments herein;

FIG. 13 depicts a composite image of the 3D bite model and the 2D imageof the patient, in accordance with one or more embodiments herein;

FIG. 13A depicts a method for extracting mouth and teeth contours from apatient's two-dimensional image and aligning with contours of a 3D modelof the patients teeth, in accordance with one or more embodimentsherein;

FIG. 13B depicts a method for determine facial landmarks, in accordancewith one or more embodiments herein;

FIG. 13C depicts a method of determining dental contours, in accordancewith one or more embodiments herein;

FIG. 13D depicts a method of determining lip contours and mouthopenings, in accordance with one or more embodiments herein;

FIG. 13E depicts alignment of dental and lip contours with a 2D image ofa patient, in accordance with one or more embodiments herein;

FIG. 13F depicts a morphing process of a facial image of patient betweena pre-treatment state and a predicted post treatment state, inaccordance with one or more embodiments herein;

FIG. 14 depicts the selection of reference points on the 2D image of apatient, in accordance with one or more embodiments herein;

FIG. 15 depicts the selection of reference points on the 3D bite modelof the patient's teeth, in accordance with one or more embodimentsherein;

FIG. 16A depicts a method for planning the treatment of a patient'steeth, in accordance with one or more embodiments herein;

FIG. 16B depicts a method for virtually representing a treatment outcomeusing automated detection of facial and dental reference objects,according to some embodiments.

FIG. 17 depicts a method of selecting and determining reference pointsand lines on an image of a patient, in accordance with one or moreembodiments herein;

FIG. 18 depicts a method of selecting and determining reference pointsand lines on an image of a patient, in accordance with one or moreembodiments herein;

FIG. 19 depicts a method of selecting and determining reference pointsand lines on an image of a patient, in accordance with one or moreembodiments herein;

FIG. 20 depicts a method of selecting and determining reference pointsand lines on an image of a patient, in accordance with one or moreembodiments herein;

FIG. 21 depicts a method of selecting and determining reference pointsand lines on a 3D bite model and an image of a patient, in accordancewith one or more embodiments herein;

FIG. 22 depicts a method of selecting and determining reference pointsand lines on an image of a patient, in accordance with one or moreembodiments herein;

FIG. 23 depicts a method determining face and head shape of a patient,in accordance with one or more embodiments herein;

FIG. 23A depicts a method of aligning a 3D model of a patient's teethwith a facial image of the patient, in accordance with one or moreembodiments herein;

FIG. 23B depicts facial landmarks from a facial image of a patient, inaccordance with one or more embodiments herein;

FIG. 23C shows a process of determining a facial midline from a facialimage of a patient, in accordance with one or more embodiments herein;

FIG. 23D shows a process of aligning a 3D model of a patient's teethwith a facial midline, in accordance with one or more embodimentsherein;

FIG. 24 depicts a method of selecting and determining reference pointsand lines on a 3D bite model of a patient, in accordance with one ormore embodiments herein;

FIG. 25 depicts various possible shapes of teeth of a patient, inaccordance with one or more embodiments herein;

FIG. 26 depicts a method determining a gingival line of a patient, inaccordance with one or more embodiments herein;

FIG. 27 depicts the placement of restorative objects in a 3D bite modelof a patient, in accordance with one or more embodiments herein;

FIG. 28 depicts the determination of the smile arc of a patient, inaccordance with one or more embodiments herein;

FIG. 29 depicts a method of measuring tooth inclination of a patient, inaccordance with one or more embodiments herein;

FIG. 30 depicts a method of determining the target final position of apatient's teeth, in accordance with one or more embodiments herein;

FIG. 31 depicts a 3D bite model integrated into a facial image a targetfinal position, in accordance with one or more embodiments herein;

FIG. 32 illustrates a composite patient image, in accordance with one ormore embodiments herein;

FIG. 33 illustrates a controller for adjusting the features of an image,in accordance with one or more embodiments herein;

FIG. 34 illustrates a 3D bite model in the composite view, in accordancewith one or more embodiments herein;

FIG. 35 illustrates a method of matching the 3D bite model with areference image, in accordance with one or more embodiments herein;

FIG. 36 illustrates a method of matching the 3D bite model with areference image, in accordance with one or more embodiments herein;

FIG. 36A depicts a process of rendering a realistic composite image ofpatient's face and a model of a patient's teeth, in accordance with oneor more embodiments herein;

FIG. 36B depicts a process of rendering a realistic composite image ofpatient's face and a model of a patient's teeth, in accordance with oneor more embodiments herein;

FIG. 37 illustrates the tooth mass reduction for restorative objects, inaccordance with one or more embodiments herein;

FIG. 38 depicts a method of determining of tooth mass low reduction, inaccordance with one or more embodiments herein;

FIG. 39 depicts a 3D bite model with restorative objects, in accordancewith one or more embodiments herein;

FIG. 40 depicts a method of forming a composite image, in accordancewith one or more embodiments herein;

FIG. 41 depicts a method for forming a composite image, in accordancewith one or more embodiments herein;

FIG. 42 depicts a portion of a method for determining tooth dimensions,in accordance with one or more embodiments herein;

FIG. 43 depicts a portion of a method for determining tooth dimensions,in accordance with one or more embodiments herein; and

FIG. 44 depicts a portion of a method for determining tooth dimensions,in accordance with one or more embodiments herein;

FIG. 45 depicts a process of rendering a realistic composite image ofpatient's face and a model of a patient's teeth, in accordance with oneor more embodiments herein.

FIG. 46 depicts a process of rendering a realistic composite image ofpatient's face and a model of a patient's teeth, in accordance with oneor more embodiments herein;

FIG. 47 depicts a process of brightening and whitening a rendering ofteeth, in accordance with one or more embodiments herein

FIG. 48 depicts a process of adding simulated specular highlights to arendering of teeth, in accordance with one or more embodiments herein

DETAILED DESCRIPTION

A better understanding of the features and advantages of the presentdisclosure will be obtained by reference to the following detaileddescription that sets forth illustrative embodiments, in which theprinciples of embodiments of the present disclosure are utilized, andthe accompanying drawings.

Although the detailed description contains many specifics, these shouldnot be construed as limiting the scope of the disclosure but merely asillustrating different examples and aspects of the present disclosure.It should be appreciated that the scope of the disclosure includes otherembodiments not discussed in detail above. Various other modifications,changes and variations which will be apparent to those skilled in theart may be made in the arrangement, operation and details of themethods, systems, and apparatus of the present disclosure providedherein without departing from the spirit and scope of the invention asdescribed herein.

As used herein the terms “dental appliance,” and “tooth receivingappliance” are treated synonymously. As used herein, a “dentalpositioning appliance” or an “orthodontic appliance” may be treatedsynonymously, and may include any dental appliance configured to changethe position of a patient's teeth in accordance with a plan, such as anorthodontic treatment plan. A “dental positioning appliance” or“orthodontic appliance,” as used herein, may include a set of dentalappliances configured to incrementally change the position of apatient's teeth over time. As noted herein, dental positioningappliances and/or orthodontic appliances may comprise polymericappliances configured to move a patient's teeth in accordance with anorthodontic treatment plan.

As used herein the term “and/or” may be used as a functional word toindicate that two words or expressions are to be taken together orindividually. For example, the phrase “A and/or B” encompasses A alone,B alone, and A and B together. Depending on context, the term “or” neednot exclude one of a plurality of words/expressions. As an example, thephrase “A or B” need not exclude A and B together.

As used herein the terms “torque” and “moment” are treated synonymously.

As used herein a “moment” may encompass a force acting on an object suchas a tooth at a distance from a center of resistance. The moment may becalculated with a vector cross product of a vector force applied to alocation corresponding to a displacement vector from the center ofresistance, for example. The moment may comprise a vector pointing in adirection. A moment opposing another moment may encompass one of themoment vectors oriented toward a first side of the object such as thetooth and the other moment vector oriented toward an opposite side ofthe object such as tooth, for example. Any discussion herein referringto application of forces on a patient's teeth is equally applicable toapplication of moments on the teeth, and vice-versa.

As used herein a “plurality of teeth” may encompass two or more teeth. Aplurality of teeth may, but need not, comprise adjacent teeth. In someembodiments, one or more posterior teeth comprises one or more of amolar, a premolar or a canine, and one or more anterior teeth comprisingone or more of a central incisor, a lateral incisor, a cuspid, a firstbicuspid or a second bicuspid.

The embodiments disclosed herein may be well suited for moving one ormore teeth of the first group of one or more teeth or moving one or moreof the second group of one or more teeth, and combinations thereof.

The embodiments disclosed herein may be well suited for combination withone or more commercially available tooth moving components such asattachments and polymeric shell appliances. In some embodiments, theappliance and one or more attachments are configured to move one or moreteeth along a tooth movement vector comprising six degrees of freedom,in which three degrees of freedom are rotational and three degrees offreedom are translation.

The present disclosure provides orthodontic appliances and relatedsystems, methods, and devices. Repositioning of teeth may beaccomplished with the use of a series of removable elastic positioningappliances such as the Invisalign® system available from AlignTechnology, Inc., the assignee of the present disclosure. Suchappliances may have a thin shell of elastic material that generallyconforms to a patient's teeth but is slightly out of alignment with aninitial or immediately prior tooth configuration. Placement of theappliance over the teeth applies controlled forces in specific locationsto gradually move the teeth into the new configuration. Repetition ofthis process with successive appliances comprising new configurationseventually moves the teeth through a series of intermediateconfigurations or alignment patterns to a final desired configuration.Repositioning of teeth may be accomplished through other series ofremovable orthodontic and/or dental appliances, including polymericshell appliances.

Although reference is made to an appliance comprising a polymeric shellappliance, the embodiments disclosed herein are well suited for use withmany appliances that receive teeth, for example appliances without oneor more of polymers or shells. The appliance can be fabricated with oneor more of many materials such as metal, glass, reinforced fibers,carbon fiber, composites, reinforced composites, aluminum, biologicalmaterials, and combinations thereof for example. The appliance can beshaped in many ways, such as with thermoforming or direct fabrication asdescribed herein, for example. Alternatively or in combination, theappliance can be fabricated with machining such as an appliancefabricated from a block of material with computer numeric controlmachining. Additionally, though reference is made herein to orthodonticappliances, at least some of the techniques described herein may applyto restorative and/or other dental appliances, including withoutlimitation crowns, veneers, teeth-whitening appliances, teeth-protectiveappliances, etc.

Turning now to the drawings, in which like numbers designate likeelements in the various figures, FIG. 1A illustrates an exemplary toothrepositioning appliance or aligner 100 that can be worn by a patient inorder to achieve an incremental repositioning of individual teeth 102 inthe jaw. The appliance can include a shell (e.g., a continuous polymericshell or a segmented shell) having teeth-receiving cavities that receiveand resiliently reposition the teeth. An appliance or portion(s) thereofmay be indirectly fabricated using a physical model of teeth. Forexample, an appliance (e.g., polymeric appliance) can be formed using aphysical model of teeth and a sheet of suitable layers of polymericmaterial. The physical model (e.g., physical mold) of teeth can beformed through a variety of techniques, including 3D printing. Theappliance can be formed by thermoforming the appliance over the physicalmodel. In some embodiments, a physical appliance is directly fabricated,e.g., using additive manufacturing techniques, from a digital model ofan appliance. In some embodiments, the physical appliance may be createdthrough a variety of direct formation techniques, such as 3D printing.An appliance can fit over all teeth present in an upper or lower jaw, orless than all of the teeth. The appliance can be designed specificallyto accommodate the teeth of the patient (e.g., the topography of thetooth-receiving cavities matches the topography of the patient's teeth),and may be fabricated based on positive or negative models of thepatient's teeth generated by impression, scanning, and the like.Alternatively, the appliance can be a generic appliance configured toreceive the teeth, but not necessarily shaped to match the topography ofthe patient's teeth. In some cases, only certain teeth received by anappliance will be repositioned by the appliance while other teeth canprovide a base or anchor region for holding the appliance in place as itapplies force against the tooth or teeth targeted for repositioning. Insome cases, some or most, and even all, of the teeth will berepositioned at some point during treatment. Teeth that are moved canalso serve as a base or anchor for holding the appliance as it is wornby the patient. In some embodiments, no wires or other means will beprovided for holding an appliance in place over the teeth. In somecases, however, it may be desirable or necessary to provide individualattachments or other anchoring elements 104 on teeth 102 withcorresponding receptacles or apertures 106 in the appliance 100 so thatthe appliance can apply a selected force on the tooth. Exemplaryappliances, including those utilized in the Invisalign® System, aredescribed in numerous patents and patent applications assigned to AlignTechnology, Inc. including, for example, in U.S. Pat. Nos. 6,450,807,and 5,975,893, as well as on the company's website, which is accessibleon the World Wide Web (see, e.g., the url “invisalign.com”). Examples oftooth-mounted attachments suitable for use with orthodontic appliancesare also described in patents and patent applications assigned to AlignTechnology, Inc., including, for example, U.S. Pat. Nos. 6,309,215 and6,830,450.

Optionally, in cases involving more complex movements or treatmentplans, it may be beneficial to utilize auxiliary components (e.g.,features, accessories, structures, devices, components, and the like) inconjunction with an orthodontic appliance. Examples of such accessoriesinclude but are not limited to elastics, wires, springs, bars, archexpanders, palatal expanders, twin blocks, occlusal blocks, bite ramps,mandibular advancement splints, bite plates, pontics, hooks, brackets,headgear tubes, springs, bumper tubes, palatal bars, frameworks,pin-and-tube apparatuses, buccal shields, buccinator bows, wire shields,lingual flanges and pads, lip pads or bumpers, protrusions, divots, andthe like. In some embodiments, the appliances, systems and methodsdescribed herein include improved orthodontic appliances with integrallyformed features that are shaped to couple to such auxiliary components,or that replace such auxiliary components.

FIG. 1B illustrates a tooth repositioning system 110 including aplurality of appliances 112, 114, 116. Any of the appliances describedherein can be designed and/or provided as part of a set of a pluralityof appliances used in a tooth repositioning system. Each appliance maybe configured so a tooth-receiving cavity has a geometry correspondingto an intermediate or final tooth arrangement intended for theappliance. The patient's teeth can be progressively repositioned from aninitial tooth arrangement towards a target tooth arrangement by placinga series of incremental position adjustment appliances over thepatient's teeth. For example, the tooth repositioning system 110 caninclude a first appliance 112 corresponding to an initial tootharrangement, one or more intermediate appliances 114 corresponding toone or more intermediate arrangements, and a final appliance 116corresponding to a target arrangement. A target tooth arrangement can bea planned final tooth arrangement selected for the patient's teeth atthe end of all planned orthodontic treatment. Alternatively, a targetarrangement can be one of some intermediate arrangements for thepatient's teeth during the course of orthodontic treatment, which mayinclude various different treatment scenarios, including, but notlimited to, instances where surgery is recommended, where interproximalreduction (IPR) is appropriate, where a progress check is scheduled,where anchor placement is best, where palatal expansion is desirable,where restorative dentistry is involved (e.g., inlays, onlays, crowns,bridges, implants, veneers, and the like), etc. As such, it isunderstood that a target tooth arrangement can be any planned resultingarrangement for the patient's teeth that follows one or more incrementalrepositioning stages. Likewise, an initial tooth arrangement can be anyinitial arrangement for the patient's teeth that is followed by one ormore incremental repositioning stages.

FIG. 1C illustrates a method 150 of orthodontic treatment using aplurality of appliances, in accordance with embodiments. The method 150can be practiced using any of the appliances or appliance sets describedherein. In block 160, a first orthodontic appliance is applied to apatient's teeth in order to reposition the teeth from a first tootharrangement to a second tooth arrangement. In block 170, a secondorthodontic appliance is applied to the patient's teeth in order toreposition the teeth from the second tooth arrangement to a third tootharrangement. The method 150 can be repeated as necessary using anysuitable number and combination of sequential appliances in order toincrementally reposition the patient's teeth from an initial arrangementto a target arrangement. The appliances can be generated all at the samestage or in sets or batches (at the beginning of a stage of thetreatment, at an intermediate stage of treatment, etc.), or theappliances can be fabricated one at a time, and the patient can weareach appliance until the pressure of each appliance on the teeth can nolonger be felt or until the maximum amount of expressed tooth movementfor that given stage has been achieved. A plurality of differentappliances (e.g., a set) can be designed and even fabricated prior tothe patient wearing any appliance of the plurality. After wearing anappliance for an appropriate period of time, the patient can replace thecurrent appliance with the next appliance in the series until no moreappliances remain. The appliances are generally not affixed to the teethand the patient may place and replace the appliances at any time duringthe procedure (e.g., patient-removable appliances). The final applianceor several appliances in the series may have a geometry or geometriesselected to overcorrect the tooth arrangement. For instance, one or moreappliances may have a geometry that would (if fully achieved) moveindividual teeth beyond the tooth arrangement that has been selected asthe “final.” Such over-correction may be desirable in order to offsetpotential relapse after the repositioning method has been terminated(e.g., permit movement of individual teeth back toward theirpre-corrected positions). Over-correction may also be beneficial tospeed the rate of correction (e.g., an appliance with a geometry that ispositioned beyond a desired intermediate or final position may shift theindividual teeth toward the position at a greater rate). In such cases,the use of an appliance can be terminated before the teeth reach thepositions defined by the appliance. Furthermore, over-correction may bedeliberately applied in order to compensate for any inaccuracies orlimitations of the appliance.

The various embodiments of the orthodontic appliances presented hereincan be fabricated in a wide variety of ways. In some embodiments, theorthodontic appliances herein (or portions thereof) can be producedusing direct fabrication, such as additive manufacturing techniques(also referred to herein as “3D printing) or subtractive manufacturingtechniques (e.g., milling). In some embodiments, direct fabricationinvolves forming an object (e.g., an orthodontic appliance or a portionthereof) without using a physical template (e.g., mold, mask etc.) todefine the object geometry. Additive manufacturing techniques can becategorized as follows: (1) vat photopolymerization (e.g.,stereolithography), in which an object is constructed layer by layerfrom a vat of liquid photopolymer resin; (2) material jetting, in whichmaterial is jetted onto a build platform using either a continuous ordrop on demand (DOD) approach; (3) binder jetting, in which alternatinglayers of a build material (e.g., a powder-based material) and a bindingmaterial (e.g., a liquid binder) are deposited by a print head; (4)fused deposition modeling (FDM), in which material is drawn though anozzle, heated, and deposited layer by layer; (5) powder bed fusion,including but not limited to direct metal laser sintering (DMLS),electron beam melting (EBM), selective heat sintering (SHS), selectivelaser melting (SLM), and selective laser sintering (SLS); (6) sheetlamination, including but not limited to laminated object manufacturing(LOM) and ultrasonic additive manufacturing (UAM); and (7) directedenergy deposition, including but not limited to laser engineering netshaping, directed light fabrication, direct metal deposition, and 3Dlaser cladding. For example, stereolithography can be used to directlyfabricate one or more of the appliances herein. In some embodiments,stereolithography involves selective polymerization of a photosensitiveresin (e.g., a photopolymer) according to a desired cross-sectionalshape using light (e.g., ultraviolet light). The object geometry can bebuilt up in a layer-by-layer fashion by sequentially polymerizing aplurality of object cross-sections. As another example, the appliancesherein can be directly fabricated using selective laser sintering. Insome embodiments, selective laser sintering involves using a laser beamto selectively melt and fuse a layer of powdered material according to adesired cross-sectional shape in order to build up the object geometry.As yet another example, the appliances herein can be directly fabricatedby fused deposition modeling. In some embodiments, fused depositionmodeling involves melting and selectively depositing a thin filament ofthermoplastic polymer in a layer-by-layer manner in order to form anobject. In yet another example, material jetting can be used to directlyfabricate the appliances herein. In some embodiments, material jettinginvolves jetting or extruding one or more materials onto a build surfacein order to form successive layers of the object geometry.

In some embodiments, the direct fabrication methods provided hereinbuild up the object geometry in a layer-by-layer fashion, withsuccessive layers being formed in discrete build steps. Alternatively orin combination, direct fabrication methods that allow for continuousbuild-up of an object's geometry can be used, referred to herein as“continuous direct fabrication.” Various types of continuous directfabrication methods can be used. As an example, in some embodiments, theappliances herein are fabricated using “continuous liquid interphaseprinting,” in which an object is continuously built up from a reservoirof photopolymerizable resin by forming a gradient of partially curedresin between the building surface of the object and apolymerization-inhibited “dead zone.”

In some embodiments, a semi-permeable membrane is used to controltransport of a photopolymerization inhibitor (e.g., oxygen) into thedead zone in order to form the polymerization gradient. Continuousliquid interphase printing can achieve fabrication speeds about 25 timesto about 100 times faster than other direct fabrication methods, andspeeds about 1000 times faster can be achieved with the incorporation ofcooling systems. Continuous liquid interphase printing is described inU.S. Patent Publication Nos. 2015/0097315, 2015/0097316, and2015/0102532, (corresponding to U.S. patent Nos. corresponding to U.S.Pat. Nos. 9,205,601, 9,216,546, and 9,211,678) the disclosures of eachof which are incorporated herein by reference in their entirety.

As another example, a continuous direct fabrication method can achievecontinuous build-up of an object geometry by continuous movement of thebuild platform (e.g., along the vertical or Z-direction) during theirradiation phase, such that the hardening depth of the irradiatedphotopolymer is controlled by the movement speed. Accordingly,continuous polymerization of material on the build surface can beachieved. Such methods are described in U.S. Pat. No. 7,892,474, thedisclosure of which is incorporated herein by reference in its entirety.

In another example, a continuous direct fabrication method can involveextruding a composite material composed of a curable liquid materialsurrounding a solid strand. The composite material can be extruded alonga continuous 3D path in order to form the object. Such methods aredescribed in U.S. Patent Publication No. 2014/0061974, corresponding toU.S. Pat. No. 9,511,543, the disclosures of which are incorporatedherein by reference in its entirety.

In yet another example, a continuous direct fabrication method utilizesa “heliolithography” approach in which the liquid photopolymer is curedwith focused radiation while the build platform is continuously rotatedand raised. Accordingly, the object geometry can be continuously builtup along a spiral build path. Such methods are described in U.S. PatentPublication No. 2014/0265034, corresponding to U.S. Pat. No. 9,321,215,the disclosures of which are incorporated herein by reference in itsentirety.

The direct fabrication approaches provided herein are compatible with awide variety of materials, including but not limited to one or more ofthe following: polymer matrix reinforced with ceramic or metallicpolymers, a polyester, a co-polyester, a polycarbonate, a thermoplasticpolyurethane, a polypropylene, a polyethylene, a polypropylene andpolyethylene copolymer, an acrylic, a cyclic block copolymer, apolyetheretherketone, a polyamide, a polyethylene terephthalate, apolybutylene terephthalate, a polyetherimide, a polyethersulfone, apolytrimethylene terephthalate, a styrenic block copolymer (SBC), asilicone rubber, an elastomeric alloy, a thermoplastic elastomer (TPE),a thermoplastic vulcanizate (TPV) elastomer, a polyurethane elastomer, ablock copolymer elastomer, a polyolefin blend elastomer, a thermoplasticco-polyester elastomer, a thermoplastic polyamide elastomer, orcombinations thereof. The materials used for direct fabrication can beprovided in an uncured form (e.g., as a liquid, resin, powder, etc.) andcan be cured (e.g., by photopolymerization, light curing, gas curing,laser curing, crosslinking, etc.) in order to form an orthodonticappliance or a portion thereof. The properties of the material beforecuring may differ from the properties of the material after curing. Oncecured, the materials herein can exhibit sufficient strength, stiffness,durability, biocompatibility, etc. for use in an orthodontic appliance.The post-curing properties of the materials used can be selectedaccording to the desired properties for the corresponding portions ofthe appliance.

In some embodiments, relatively rigid portions of the orthodonticappliance can be formed via direct fabrication using one or more of thefollowing materials: a polyester, a co-polyester, a polycarbonate, athermoplastic polyurethane, a polypropylene, a polyethylene, apolypropylene and polyethylene copolymer, an acrylic, a cyclic blockcopolymer, a polyetheretherketone, a polyamide, a polyethyleneterephthalate, a polybutylene terephthalate, a polyetherimide, apolyethersulfone, and/or a polytrimethylene terephthalate.

In some embodiments, relatively elastic portions of the orthodonticappliance can be formed via direct fabrication using one or more of thefollowing materials: a styrenic block copolymer (SBC), a siliconerubber, an elastomeric alloy, a thermoplastic elastomer (TPE), athermoplastic vulcanizate (TPV) elastomer, a polyurethane elastomer, ablock copolymer elastomer, a polyolefin blend elastomer, a thermoplasticco-polyester elastomer, and/or a thermoplastic polyamide elastomer.

Optionally, the direct fabrication methods described herein allow forfabrication of an appliance including multiple materials, referred toherein as “multi-material direct fabrication.” In some embodiments, amulti-material direct fabrication method involves concurrently formingan object from multiple materials in a single manufacturing step usingthe same fabrication machine and method. For instance, a multi-tipextrusion apparatus can be used to selectively dispense multiple typesof materials (e.g., resins, liquids, solids, or combinations thereof)from distinct material supply sources in order to fabricate an objectfrom a plurality of different materials. Such methods are described inU.S. Pat. No. 6,749,414, the disclosure of which is incorporated hereinby reference in its entirety. Alternatively or in combination, amulti-material direct fabrication method can involve forming an objectfrom multiple materials in a plurality of sequential manufacturingsteps. For instance, a first portion of the object can be formed from afirst material in accordance with any of the direct fabrication methodsherein, then a second portion of the object can be formed from a secondmaterial in accordance with methods herein, and so on, until theentirety of the object has been formed. The relative arrangement of thefirst and second portions can be varied as desired, e.g., the firstportion can be partially or wholly encapsulated by the second portion ofthe object. The sequential manufacturing steps can be performed usingthe same fabrication machine or different fabrication machines, and canbe performed using the same fabrication method or different fabricationmethods. For example, a sequential multi-manufacturing procedure caninvolve forming a first portion of the object using stereolithographyand a second portion of the object using fused deposition modeling.

Direct fabrication can provide various advantages compared to othermanufacturing approaches. For instance, in contrast to indirectfabrication, direct fabrication permits production of an orthodonticappliance without utilizing any molds or templates for shaping theappliance, thus reducing the number of manufacturing steps involved andimproving the resolution and accuracy of the final appliance geometry.Additionally, direct fabrication permits precise control over the 3Dgeometry of the appliance, such as the appliance thickness. Complexstructures and/or auxiliary components can be formed integrally as asingle piece with the appliance shell in a single manufacturing step,rather than being added to the shell in a separate manufacturing step.In some embodiments, direct fabrication is used to produce appliancegeometries that would be difficult to create using alternativemanufacturing techniques, such as appliances with very small or finefeatures, complex geometric shapes, undercuts, interproximal structures,shells with variable thicknesses, and/or internal structures (e.g., forimproving strength with reduced weight and material usage). For example,in some embodiments, the direct fabrication approaches herein permitfabrication of an orthodontic appliance with feature sizes of less thanor equal to about 5 μm, or within a range from about 5 μm to about 50μm, or within a range from about 20 μm to about 50 μm.

In some embodiments, the direct fabrication methods described hereinimplement process controls for various machine parameters of a directfabrication system or device in order to ensure that the resultantappliances are fabricated with a high degree of precision. Suchprecision can be beneficial for ensuring accurate delivery of a desiredforce system to the teeth in order to effectively elicit toothmovements. Process controls can be implemented to account for processvariability arising from multiple sources, such as the materialproperties, machine parameters, environmental variables, and/orpost-processing parameters.

Material properties may vary depending on the properties of rawmaterials, purity of raw materials, and/or process variables duringmixing of the raw materials. In many embodiments, resins or othermaterials for direct fabrication should be manufactured with tightprocess control to ensure little variability in photo-characteristics,material properties (e.g., viscosity, surface tension), physicalproperties (e.g., modulus, strength, elongation) and/or thermalproperties (e.g., glass transition temperature, heat deflectiontemperature). Process control for a material manufacturing process canbe achieved with screening of raw materials for physical propertiesand/or control of temperature, humidity, and/or other process parametersduring the mixing process. By implementing process controls for thematerial manufacturing procedure, reduced variability of processparameters and more uniform material properties for each batch ofmaterial can be achieved. Residual variability in material propertiescan be compensated with process control on the machine, as discussedfurther herein.

Machine parameters can include curing parameters. For digital lightprocessing (DLP)-based curing systems, curing parameters can includepower, curing time, and/or grayscale of the full image. For laser-basedcuring systems, curing parameters can include power, speed, beam size,beam shape and/or power distribution of the beam. For printing systems,curing parameters can include material drop size, viscosity, and/orcuring power. These machine parameters can be monitored and adjusted ona regular basis (e.g., some parameters at every 1-x layers and someparameters after each build) as part of the process control on thefabrication machine. Process control can be achieved by including asensor on the machine that measures power and other beam parametersevery layer or every few seconds and automatically adjusts them with afeedback loop. For DLP machines, gray scale can be measured andcalibrated before, during, and/or at the end of each build, and/or atpredetermined time intervals (e.g., every n^(th) build, once per hour,once per day, once per week, etc.), depending on the stability of thesystem. In addition, material properties and/or photo-characteristicscan be provided to the fabrication machine, and a machine processcontrol module can use these parameters to adjust machine parameters(e.g., power, time, gray scale, etc.) to compensate for variability inmaterial properties. By implementing process controls for thefabrication machine, reduced variability in appliance accuracy andresidual stress can be achieved.

In many embodiments, environmental variables (e.g., temperature,humidity, Sunlight or exposure to other energy/curing source) aremaintained in a tight range to reduce variable in appliance thicknessand/or other properties. Optionally, machine parameters can be adjustedto compensate for environmental variables.

In many embodiments, post-processing of appliances includes cleaning,post-curing, and/or support removal processes. Relevant post-processingparameters can include purity of cleaning agent, cleaning pressureand/or temperature, cleaning time, post-curing energy and/or time,and/or consistency of support removal process. These parameters can bemeasured and adjusted as part of a process control scheme. In addition,appliance physical properties can be varied by modifying thepost-processing parameters. Adjusting post-processing machine parameterscan provide another way to compensate for variability in materialproperties and/or machine properties.

Although various embodiments herein are described with respect to directfabrication techniques, it shall be appreciated that other techniquescan also be used, such as indirect fabrication techniques. In someembodiments, the appliances herein (or portions thereof) can be producedusing indirect fabrication techniques, such as by thermoforming over apositive or negative mold. Indirect fabrication of an orthodonticappliance can involve one or more of the following steps: producing apositive or negative mold of the patient's dentition in a targetarrangement (e.g., by additive manufacturing, milling, etc.),thermoforming one or more sheets of material over the mold in order togenerate an appliance shell, forming one or more structures in the shell(e.g., by cutting, etching, etc.), and/or coupling one or morecomponents to the shell (e.g., by extrusion, additive manufacturing,spraying, thermoforming, adhesives, bonding, fasteners, etc.).Optionally, one or more auxiliary appliance components as describedherein (e.g., elastics, wires, springs, bars, arch expanders, palatalexpanders, twin blocks, occlusal blocks, bite ramps, mandibularadvancement splints, bite plates, pontics, hooks, brackets, headgeartubes, bumper tubes, palatal bars, frameworks, pin-and-tube apparatuses,buccal shields, buccinator bows, wire shields, lingual flanges and pads,lip pads or bumpers, protrusions, divots, etc.) are formed separatelyfrom and coupled to the appliance shell (e.g., via adhesives, bonding,fasteners, mounting features, etc.) after the shell has been fabricated.

In some embodiments, the orthodontic appliances herein can be fabricatedusing a combination of direct and indirect fabrication techniques, suchthat different portions of an appliance can be fabricated usingdifferent fabrication techniques and assembled in order to form thefinal appliance. For example, an appliance shell can be formed byindirect fabrication (e.g., thermoforming), and one or more structuresor components as described herein (e.g., auxiliary components, powerarms, etc.) can be added to the shell by direct fabrication (e.g.,printing onto the shell).

The configuration of the orthodontic appliances herein can be determinedaccording to a treatment plan for a patient, e.g., a treatment planinvolving successive administration of a plurality of appliances forincrementally repositioning teeth. Computer-based treatment planningand/or appliance manufacturing methods can be used in order tofacilitate the design and fabrication of appliances. For instance, oneor more of the appliance components described herein can be digitallydesigned and fabricated with the aid of computer-controlledmanufacturing devices (e.g., computer numerical control (CNC) milling,computer-controlled additive manufacturing such as 3D printing, etc.).The computer-based methods presented herein can improve the accuracy,flexibility, and convenience of appliance fabrication.

In some embodiments, computer-based 3D planning/design tools, such asTreat™ software from Align Technology, Inc., may be used to design andfabricate the orthodontic appliances described herein.

FIG. 2 illustrates a method 200 for designing an orthodontic applianceto be fabricated, in accordance with embodiments. The method 200 can beapplied to any embodiment of the orthodontic appliances describedherein. Some or all of the operations of the method 200 can be performedby any suitable data processing system or device, e.g., one or moreprocessors configured with suitable instructions.

In block 210, a movement path to move one or more teeth from an initialarrangement to a target arrangement is determined. The initialarrangement can be determined from a mold or a scan of the patient'steeth or mouth tissue, e.g., using wax bites, direct contact scanning,x-ray imaging, tomographic imaging, sonographic imaging, and othertechniques for obtaining information about the position and structure ofthe teeth, jaws, gums and other orthodontically relevant tissue. Fromthe obtained data, a digital data set can be derived that represents theinitial (e.g., pretreatment) arrangement of the patient's teeth andother tissues. Optionally, the initial digital data set is processed tosegment the tissue constituents from each other. For example, datastructures that digitally represent individual tooth crowns can beproduced. Advantageously, digital models of entire teeth can beproduced, including measured or extrapolated hidden surfaces and rootstructures, as well as surrounding bone and soft tissue.

The target arrangement of the teeth (e.g., a desired and intended endresult of orthodontic treatment) can be received from a clinician in theform of a prescription, can be calculated from basic orthodonticprinciples, and/or can be extrapolated computationally from a clinicalprescription. With a specification of the desired final positions of theteeth and a digital representation of the teeth themselves, the finalposition and surface geometry of each tooth can be specified to form acomplete model of the tooth arrangement at the desired end of treatment.

Having both an initial position and a target position for each tooth, amovement path can be defined for the motion of each tooth. In someembodiments, the movement paths are configured to move the teeth in thequickest fashion with the least amount of round-tripping to bring theteeth from their initial positions to their desired target positions.The tooth paths can optionally be segmented, and the segments can becalculated so that each tooth's motion within a segment stays withinthreshold limits of linear and rotational translation. In this way, theend points of each path segment can constitute a clinically viablerepositioning, and the aggregate of segment end points can constitute aclinically viable sequence of tooth positions, so that moving from onepoint to the next in the sequence does not result in a collision ofteeth.

In block 220, a force system to produce movement of the one or moreteeth along the movement path is determined. A force system can includeone or more forces and/or one or more torques. Different force systemscan result in different types of tooth movement, such as tipping,translation, rotation, extrusion, intrusion, root movement, etc.Biomechanical principles, modeling techniques, forcecalculation/measurement techniques, and the like, including knowledgeand approaches commonly used in orthodontia, may be used to determinethe appropriate force system to be applied to the tooth to accomplishthe tooth movement. In determining the force system to be applied,sources may be considered including literature, force systems determinedby experimentation or virtual modeling, computer-based modeling,clinical experience, minimization of unwanted forces, etc.

Determination of the force system can be performed in a variety of ways.For example, in some embodiments, the force system is determined on apatient-by-patient basis, e.g., using patient-specific data.Alternatively or in combination, the force system can be determinedbased on a generalized model of tooth movement (e.g., based onexperimentation, modeling, clinical data, etc.), such thatpatient-specific data is not necessarily used. In some embodiments,determination of a force system involves calculating specific forcevalues to be applied to one or more teeth to produce a particularmovement. Alternatively, determination of a force system can beperformed at a high level without calculating specific force values forthe teeth. For instance, block 220 can involve determining a particulartype of force to be applied (e.g., extrusive force, intrusive force,translational force, rotational force, tipping force, torqueing force,etc.) without calculating the specific magnitude and/or direction of theforce.

In block 230, an appliance geometry and/or material composition for anorthodontic appliance configured to produce the force system isdetermined. The appliance can be any embodiment of the appliancesdiscussed herein, such as an appliance having variable localizedproperties, integrally formed components, and/or power arms.

For example, in some embodiments, the appliance comprises aheterogeneous thickness, a heterogeneous stiffness, or a heterogeneousmaterial composition. In some embodiments, the appliance comprises twoor more of a heterogeneous thickness, a heterogeneous stiffness, or aheterogeneous material composition. In some embodiments, the appliancecomprises a heterogeneous thickness, a heterogeneous stiffness, and aheterogeneous material composition. The heterogeneous thickness,stiffness, and/or material composition can be configured to produce theforce system for moving the teeth, e.g., by preferentially applyingforces at certain locations on the teeth. For example, an appliance withheterogeneous thickness can include thicker portions that apply moreforce on the teeth than thinner portions. As another example, anappliance with heterogeneous stiffness can include stiffer portions thatapply more force on the teeth than more elastic portions. Variations instiffness can be achieved by varying the appliance thickness, materialcomposition, and/or degree of photopolymerization, as described herein.

In some embodiments, determining the appliance geometry and/or materialcomposition comprises determining the geometry and/or materialcomposition of one or more integrally formed components to be directlyfabricated with an appliance shell. The integrally formed component canbe any of the embodiments described herein. The geometry and/or materialcomposition of the integrally formed component(s) can be selected tofacilitate application of the force system onto the patient's teeth. Thematerial composition of the integrally formed component can be the sameas or different from the material composition of the shell.

In some embodiments, determining the appliance geometry comprisesdetermining the geometry for a variable gable bend.

The block 230 can involve analyzing the desired force system in order todetermine an appliance geometry and material composition that wouldproduce the force system. In some embodiments, the analysis involvesdetermining appliance properties (e.g., stiffness) at one or morelocations that would produce a desired force at the one or morelocations. The analysis can then involve determining an appliancegeometry and material composition at the one or more locations toachieve the specified properties. Determination of the appliancegeometry and material composition can be performed using a treatment orforce application simulation environment. A simulation environment caninclude, e.g., computer modeling systems, biomechanical systems orapparatus, and the like. Optionally, digital models of the applianceand/or teeth can be produced, such as finite element models. The finiteelement models can be created using computer program applicationsoftware available from a variety of vendors. For creating solidgeometry models, computer aided engineering (CAE) or computer aideddesign (CAD) programs can be used, such as the AutoCAD® softwareproducts available from Autodesk, Inc., of San Rafael, Calif. Forcreating finite element models and analyzing them, program products froma number of vendors can be used, including finite element analysispackages from ANSYS, Inc., of Canonsburg, Pa., and SIMULIA (Abaqus)software products from Dassault Systèmes of Waltham, Mass.

Optionally, one or more appliance geometries and material compositionscan be selected for testing or force modeling. As noted above, a desiredtooth movement, as well as a force system required or desired foreliciting the desired tooth movement, can be identified. Using thesimulation environment, a candidate appliance geometry and compositioncan be analyzed or modeled for determination of an actual force systemresulting from use of the candidate appliance. One or more modificationscan optionally be made to a candidate appliance, and force modeling canbe further analyzed as described, e.g., in order to iterativelydetermine an appliance design that produces the desired force system.

Optionally, block 230 can further involve determining the geometry ofone or more auxiliary components to be used in combination with theorthodontic appliance in order to exert the force system on the one ormore teeth. Such auxiliaries can include one or more of tooth-mountedattachments, elastics, wires, springs, bite blocks, arch expanders,wire-and-bracket appliances, shell appliances, headgear, or any otherorthodontic device or system that can be used in conjunction with theorthodontic appliances herein. The use of such auxiliary components maybe advantageous in situations where it is difficult for the appliancealone to produce the force system. Additionally, auxiliary componentscan be added to the orthodontic appliance in order to provide otherdesired functionalities besides producing the force system, such asmandibular advancement splints to treat sleep apnea, pontics to improveaesthetic appearance, and so on. In some embodiments, the auxiliarycomponents are fabricated and provided separately from the orthodonticappliance. Alternatively, the geometry of the orthodontic appliance canbe modified to include one or more auxiliary components as integrallyformed components.

In block 240, instructions for fabrication of the orthodontic appliancehaving the appliance geometry and material composition are generated.The instructions can be configured to control a fabrication system ordevice in order to produce the orthodontic appliance with the specifiedappliance geometry and material composition. In some embodiments, theinstructions are configured for manufacturing the orthodontic applianceusing direct fabrication (e.g., stereolithography, selective lasersintering, fused deposition modeling, 3D printing, continuous directfabrication, multi-material direct fabrication, etc.). Optionally, theinstructions can be configured to cause a fabrication machine todirectly fabricate the orthodontic appliance with teeth receivingcavities having variable gable bends, as discussed above and herein. Inalternative embodiments, the instructions can be configured for indirectfabrication of the appliance, e.g., by thermoforming.

Although the above blocks show a method 200 of designing an orthodonticappliance in accordance with some embodiments, a person of ordinaryskill in the art will recognize some variations based on the teachingdescribed herein. Some of the blocks may comprise sub-blocks. Some ofthe blocks may be repeated as often as desired. One or more blocks ofthe method 200 may be performed with any suitable fabrication system ordevice, such as the embodiments described herein. Some of the blocks maybe optional, and the order of the blocks can be varied as desired. Forinstance, in some embodiments, block 220 is optional, such that block230 involves determining the appliance geometry and/or materialcomposition based directly on the tooth movement path rather than basedon the force system.

FIG. 3 illustrates a method 300 for digitally planning an orthodontictreatment and/or design or fabrication of an appliance, in accordancewith embodiments. The method 300 can be applied to any of the treatmentprocedures described herein and can be performed by any suitable dataprocessing system.

In block 310, a digital representation of a patient's teeth is received.The digital representation can include surface topography data for thepatient's intraoral cavity (including teeth, gingival tissues, etc.).The surface topography data can be generated by directly scanning theintraoral cavity, a physical model (positive or negative) of theintraoral cavity, or an impression of the intraoral cavity, using asuitable scanning device (e.g., a handheld scanner, desktop scanner,etc.).

In block 320, one or more treatment stages are generated based on thedigital representation of the teeth. The treatment stages can beincremental repositioning stages of an orthodontic treatment proceduredesigned to move one or more of the patient's teeth from an initialtooth arrangement to a target arrangement. For example, the treatmentstages can be generated by determining the initial tooth arrangementindicated by the digital representation, determining a target tootharrangement, and determining movement paths of one or more teeth in theinitial arrangement necessary to achieve the target tooth arrangement.The movement path can be optimized based on minimizing the totaldistance moved, preventing collisions between teeth, avoiding toothmovements that are more difficult to achieve, or any other suitablecriteria.

In block 330, at least one orthodontic appliance is fabricated based onthe generated treatment stages. For example, a set of appliances can befabricated, each shaped according to a tooth arrangement specified byone of the treatment stages, such that the appliances can besequentially worn by the patient to incrementally reposition the teethfrom the initial arrangement to the target arrangement. The applianceset may include one or more of the orthodontic appliances describedherein. The fabrication of the appliance may involve creating a digitalmodel of the appliance to be used as input to a computer-controlledfabrication system. The appliance can be formed using direct fabricationmethods, indirect fabrication methods, or combinations thereof, asdesired.

In some instances, staging of various arrangements or treatment stagesmay not be necessary for design and/or fabrication of an appliance. Asillustrated by the dashed line in FIG. 3, design and/or fabrication ofan orthodontic appliance, and perhaps a particular orthodontictreatment, may include use of a representation of the patient's teeth(e.g., receive a digital representation of the patient's teeth 310),followed by design and/or fabrication of an orthodontic appliance basedon a representation of the patient's teeth in the arrangementrepresented by the received representation.

Optionally, some or all of the blocks of the method 300 are performedlocally at the site where the patient is being treated and during asingle patient visit, referred to herein as “chair side manufacturing.”Chair side manufacturing can involve, for example, scanning thepatient's teeth, automatically generating a treatment plan withtreatment stages, and immediately fabricating one or more orthodonticappliance(s) to treat the patient using a chair side direct fabricationmachine, all at the treating professional's office during a singleappointment. In embodiments where a series of appliances are used totreat the patient, the first appliance may be produced chair side forimmediate delivery to the patient, with the remaining appliancesproduced separately (e.g., off site at a lab or central manufacturingfacility) and delivered at a later time (e.g., at a follow upappointment, mailed to the patient). Alternatively, the methods hereincan accommodate production and immediate delivery of the entire seriesof appliances on site during a single visit. Chair side manufacturingcan thus improve the convenience and speed of the treatment procedure byallowing the patient to immediately begin treatment at thepractitioner's office, rather than having to wait for fabrication anddelivery of the appliances at a later date. Additionally, chair sidemanufacturing can provide improved flexibility and efficiency oforthodontic treatment. For instance, in some embodiments, the patient isre-scanned at each appointment to determine the actual positions of theteeth, and the treatment plan is updated accordingly. Subsequently, newappliances can be immediately produced and delivered chair side toaccommodate any changes to or deviations from the treatment plan.

FIG. 4 is a simplified block diagram of a data processing system 400that may be used in executing methods and processes described herein.The data processing system 400 typically includes at least one processor402 that communicates with one or more peripheral devices via bussubsystem 404. These peripheral devices typically include a storagesubsystem 406 (memory subsystem 408 and file storage subsystem 414), aset of user interface input and output devices 418, and an interface tooutside networks 416. This interface is shown schematically as “NetworkInterface” block 416, and is coupled to corresponding interface devicesin other data processing systems via communication network interface424. Data processing system 400 can include, for example, one or morecomputers, such as a personal computer, workstation, mainframe, laptop,and the like.

The user interface input devices 418 are not limited to any particulardevice, and can typically include, for example, a keyboard, pointingdevice, mouse, scanner, interactive displays, touchpad, joysticks, etc.Similarly, various user interface output devices can be employed in asystem of the invention, and can include, for example, one or more of aprinter, display (e.g., visual, non-visual) system/subsystem,controller, projection device, audio output, and the like.

Storage subsystem 406 maintains the basic required programming,including computer readable media having instructions (e.g., operatinginstructions, etc.), and data constructs. The program modules discussedherein are typically stored in storage subsystem 406. Storage subsystem406 typically includes memory subsystem 408 and file storage subsystem414. Memory subsystem 408 typically includes a number of memories (e.g.,RAM 410, ROM 412, etc.) including computer readable memory for storageof fixed instructions, instructions and data during program execution,basic input/output system, etc. File storage subsystem 414 providespersistent (non-volatile) storage for program and data files, and caninclude one or more removable or fixed drives or media, hard disk,floppy disk, CD-ROM, DVD, optical drives, and the like. One or more ofthe storage systems, drives, etc., may be located at a remote location,such coupled via a server on a network or via the internet/World WideWeb. In this context, the term “bus subsystem” is used generically so asto include any mechanism for letting the various components andsubsystems communicate with each other as intended and can include avariety of suitable components/systems that would be known or recognizedas suitable for use therein. It will be recognized that variouscomponents of the system can be, but need not necessarily be at the samephysical location, but could be connected via various local-area orwide-area network media, transmission systems, etc.

Scanner 420 includes any means for obtaining a digital representation(e.g., images, surface topography data, etc.) of a patient's teeth(e.g., by scanning physical models of the teeth such as casts 421, byscanning impressions taken of the teeth, or by directly scanning theintraoral cavity), which can be obtained either from the patient or fromtreating professional, such as an orthodontist, and includes means ofproviding the digital representation to data processing system 400 forfurther processing. Scanner 420 may be located at a location remote withrespect to other components of the system and can communicate image dataand/or information to data processing system 400, for example, via anetwork interface 424. Fabrication system 422 fabricates appliances 423based on a treatment plan, including data set information received fromdata processing system 400. Fabrication machine 422 can, for example, belocated at a remote location and receive data set information from dataprocessing system 400 via network interface 424. The camera 425 mayinclude any image capture device configured to capture still images ormovies. The camera 425 may facilitate capturing various perspectives ofa patient's dentition. In some implementations, the camera 425 mayfacilitate capture of images at various focal lengths and distances fromthe patient.

The data processing aspects of the methods described herein can beimplemented in digital electronic circuitry, or in computer hardware,firmware, software, or suitable combinations thereof. Data processingapparatus can be implemented in a computer program product tangiblyembodied in a machine-readable storage device for execution by aprogrammable processor. Data processing blocks can be performed by aprogrammable processor executing program instructions to performfunctions by operating on input data and generating output. The dataprocessing aspects can be implemented in one or more computer programsthat are executable on a programmable system, the system including oneor more programmable processors operably coupled to a data storagesystem. Generally, a processor will receive instructions and data from aread-only memory and/or a random access memory. Storage devices suitablefor tangibly embodying computer program instructions and data includeall forms of nonvolatile memory, such as: semiconductor memory devices,such as EPROM, EEPROM, and flash memory devices; magnetic disks such asinternal hard disks and removable disks; magneto-optical disks; andCD-ROM disks.

Forming a Photo-Realistic Composite Image of a Patient's Dentition

In some embodiments, one or more modules of the data processing system400 may be configured to form a highly accurate composite image of apatient's dentition. As noted further herein, the processor(s) 402 mayexecute computer-implemented instructions stored on the storagesubsystem 406 to gather a 3D model of a patient's dentition. The 3Dmodel may include 3D virtual representations of a patient's dentitionfor display on a display device. In some implementations, the 3D modelmay facilitate modifications of parameters of teeth (locations, sizes,shapes, etc.) based on application of an orthodontic treatment plan. Forinstance, the 3D model may facilitate visualization of how a patient'steeth move and/or are represented after one or more orthodontictreatment plans have been applied to the patient's teeth. Theprocessor(s) 402 may execute computer-implemented instructions stored onthe storage subsystem 406 to gather an image of the patient that, insome embodiments, includes an image (e.g., a 2D image) of the patient'sface and dentition. The image may have been captured with a camera,uploaded with a phone or computer, and/or uploaded over a network (e.g.,the Internet).

In various implementations, the user interface and output device 418 mayreceive from a technician (e.g., a treatment professional) a first setof reference points corresponding to specific locations of the patient'sdentition as represented on the 3D model of the patient's dentition. Thefirst set of reference points may correspond to annotations, markups,drawn points, etc. placed on relevant anatomical points of the 3D model.The user interface and output device 148 may further receive from thetechnician a second set of reference points on a portion of the 2D imageof the patient. In some implementations, the second set of referencepoints may correspond to annotations, markups, drawn points, etc. placedon parts of the 2D image that correspond to the relevant anatomicalpoints identified by the first set of reference points on the 3D model.

In some implementations, the processor(s) 402 may execute programinstructions stored on the storage subsystem(s) 408 that combine the 3Dmodel of the patient's dentition with the image of the patient,particularly the dentition portion of the image of the patient. Such acombination may include identification of sizes, shapes, and/orperspectives of the various teeth modeled in the 3D model and/orrepresented in the image(s) of the patient's dentition. Combination mayfurther include adjusting and/or scaling the models and/orrepresentations of teeth so that specific parts of the 3D model have thesame size/shape/perspective, etc. as similar parts of 2D images.

In some implementations, the processor(s) 402 may execute programinstructions stored on the storage subsystem(s) 408 that align the firstset of reference points on the 3D model with the second set of referencepoints on the image of the patient, particularly portions of the imageof the patient corresponding to the patient's dentition. Alignment mayinvolve adjusting the 3D model to the image so that the 3D model moreaccurately models the patient's dentition. In some implementations, thealignment involves mathematically correlating points on the 3D modelwith points on the image. As an example, the alignment may involve astatistical technique to ensure distances between the representations ofthe first set of reference points on the 3D model and therepresentations of the second set of reference points on the image areminimized and/or otherwise optimized. In various implementations, thealignment involves creating a projection plane in a 3D space using theimage. The alignment, in such implementations, may further includeminimizing the sum of squares of distances between corresponding pointsin the projection plane and the 3D model of the patient's dentition.Within the 3D model, the processor(s) 402 may execute programinstructions stored on the storage subsystem(s) 408 to rotate and/orshift the 3D model of either jaw (mandible or maxilla) of a patient whenthe alignment is performed. Additionally, the processor(s) 402 mayexecute program instructions stored on the storage subsystem(s) 408 tochange focal lengths, zooms, and/or other aspects of perspectives in the3D model to facilitate visualization of a patient's dentition.

In some implementations, the processor(s) 402 may execute programinstructions stored on the storage subsystem(s) 408 to display amodified 2D representation of a patient's dentition. In someimplementations, the modified 2D representation may include estimatedresults of an orthodontic treatment plan. Rather than providing acomputerized or otherwise non-human depiction of the application of theorthodontic treatment plan, the modified 2D representation may show ahighly photo-realistic, emotion-evocative, humanistic visualization ofhow application of an orthodontic treatment plan will appear on thepatient. Such a modified rendering may solve technical problems relatedto computerized visualization of treatment planning by use of automatedimage matching processes.

FIG. 5A depicts a method 500 a of building a composite image, inaccordance with one or more embodiments herein. The method 500 a may beused with or in the processes and/or systems described above withreference to FIGS. 1-4. A composite image, such as the composite image600 (shown in FIG. 6), can be helpful in the treatment planning processand treatment of a patient. By integrating a 3D bite model into a 2Dimage of a patient, the dental professional can evaluate the finalposition of the teeth while taking into account facial features. A 3Dbite model may be a 3D model of teeth, such as a patient's teeth. Inaddition, the treatment plan and proposed final or target position ofthe patient's teeth may be determined with the aid of the facial imageof the patient, thereby creating a treatment plan that positions theteeth based on a final orthodontic position that corrects malocclusionsand takes into account facial features, such as head shape, eyeposition, facial midline, and other facial features described herein.

To take into account the facial features of a patient, a composite imageof the patient is formed using a 2D image of the face of the patient anda 3D bite model generated from a scan of the patient's mouth or of apositive of negative model of a patient's teeth and gums. The 3D bitemodel of the patient with the teeth in an initial position is integratedinto the 2D image of the patient such that the teeth in the 3D bitemodel are placed in the same position at the teeth in the 2D image ofthe patient. At block 510 of method 500 a, a 3D bite model of thepatient's teeth is built. Such a model may be formed as described abovewith reference to FIGS. 1-4. The 3D bite model includes a model of thepatient's teeth and gingiva in an initial position, such as beforetreatment, in a final or target position according to a treatment plan,or in an interim position that depicts the teeth and gums during thetreatment process. The system 400 may build the 3D bite model.

At block 520 a 2D image of the patient is received. In some embodiments,the 2D image includes the mouth of the patient and one or more of theface, head, neck, shoulders, torso, or the entire patient. The 2D imageof the patient may include an image of the patient with their mouth inone or more positions; for example, the patient's mouth may be a smilingposition, such as a social smiling position, a repose position withrelaxed muscles and lips slightly parted, or an anterior retracted openbite or closed bite positions.

In some embodiments, at block 520 an image of the patient is taken witha digital imaging device such as the camera 425. The image may becaptured with a lens at a predetermined focal length and at a distancefrom the patient. The image may be captured remotely and then receivedfor processing, for example, by the system 400 of FIG. 4.

At block 530 reference points are selected or otherwise identified onthe 3D bite model and the 2D image of the patient. In some embodiments,the reference points may include the gingival apex of one or more teeth,such as the anterior teeth. In some embodiments, the reference pointsmay include the midpoint of the incisal edge of the teeth or the ends ofthe incisal edges of teeth. In some embodiments, the reference pointsmay be a cusp tip of one or more teeth, such as the cusp tips of thecanine teeth. In some embodiments, facial landmarks and contours,selected using neural networks, or otherwise, as descried herein, may beused as reference points. Examples of reference points are further shownin FIGS. 7, 8, 9, 14 and 15 and discussed further herein.

The references points of the 3D bite model correspond to referencepoints at the same location in the 2D image. For example, the referencepoints include the gingival apex of each of the six anterior teeth onthe 3D bite model and the reference points on the 2D image of thepatient can also include the gingival apex of each of the six anteriorteeth. Each of the reference points on a tooth or gingiva of the 3D bitemodel may correspond with a reference point on a tooth or gingiva of the2D image. For example, a reference point at the left incisal gingivalapex of the 3D bite model can correspond with the left incisal gingivalapex of the 2D image of the patient. Such two corresponding referencepoints may be considered a pair of corresponding reference points.

At block 540 the 3D bite model is combined with the 2D image of thepatient. The 3D bite model is manipulated and positioned such that theteeth of the 3D bite model are in the same position and the same size asthe teeth of the patient in the 2D image.

In some embodiments, the 3D bite model is positioned in the 2D image byaligning each of the reference points on the 3D bite model to each ofthe corresponding reference points on the 2D image of the patient. Insome embodiments, the reference points of the 3D bite model are alignedwith the corresponding reference points on the 2D image of the patientby minimizing the sum of the distance between each corresponding pair ofreference points. In some embodiments, the reference points of the 3Dbite model are aligned with the corresponding reference points on the 2Dimage of the patient by minimizing the sum of the squares of thedistance between each corresponding pair of reference points.

In some embodiments, combining the 3D bite model with the 2D image ofthe patient includes determining a mouth opening in the 2D image. Insome embodiments, the mouth opening is the shape of the inside edge ofthe patient's lips in the 2D image. In some embodiments, the portion ofthe 2D image within the mouth opening is removed or otherwise deletedfrom the 2D image of the patient. In some embodiments, the 3D bitemodel, or a 2D projection of the 3D bite model is placed or renderedbehind the 2D image such that the 3D bite model, or at least a portionof the 3D bite model, is visible through the mouth opening of the 2Dimage of the patient.

At block 550 the distance between one or more pairs of reference pointsor features is minimized. In some embodiments, the 3D bite model isrotated or translated about one or more of the three perpendicular axesof 3D space. In some embodiments, the size of the teeth relative to the2D image may be selected by matching the distance and focal length ofthe rendering of the 3D bite model with focal length and the distancebetween the imaging device and the patient used when capturing the 2Dimage of the patient.

At block 560 the composite image is formed. In some embodiments, thecomposite image is a composite 2D image that integrates the 2D image ofthe patient and a 2D rendering of the 3D bite model, or at least aportion of the 3D bite model, viewable through the mouth opening of the2D image. In some embodiments, the composite image is formed bydisplaying a 3D rendering of the 3D bite model, or at least a portion ofthe 3D bite model, behind and viewable through the mouth opening of the2D image of the patient on a surface or on a monitor. In variousimplementations, the composite image comprises a modified 2Drepresentation of a patient's dentition. In some implementations, themodified 2D representation may include estimated results of anorthodontic treatment plan. As noted herein, rather than providing acomputerized or otherwise non-human depiction of the application of theorthodontic treatment plan, the modified 2D representation may show ahighly photo-realistic, emotion-evocative, humanistic visualization ofhow application of an orthodontic treatment plan will appear on thepatient. Such a modified rendering may solve technical problems relatedto computerized visualization of treatment planning by use of automatedimage matching processes.

At block 570, the composite image is displayed. The composite image maybe displayed on a display device accessible to a patient locally,remotely, etc. In some implementations, the composite image is displayedremotely to the patient over a network connection (e.g., an Internetconnection) on a digital device of the patient. The composite image maydisplayed in substantial real-time (e.g., while the patient is in theoffice and getting images of their dentition taken) or after a visit. Insome implementations, the composite image is rendered into a format thatcan be displayed on a standalone application, a web page, a mobileapplication, and/or other portal for the patient.

FIG. 5B depicts a method 500 b of building a composite image, inaccordance with one or more embodiments herein. The method 500 b may beused with or in the processes and/or systems described above withreference to FIGS. 1-4. A composite image, such as the composite image600 (shown in FIG. 6), can be helpful in the treatment planning processand treatment of a patient. At an operation 582, a three-dimensional ofthe patient's dentition is gathered. The 3D model may comprise a virtualrepresentation of the patient's dentition at a specific treatment stageof an orthodontic treatment plan.

At an operation 584, an image of the patient is gathered. In someimplementations, the image includes at least a portion of the patient'sface and including at least a portion of the patient's dentition. At anoperation 586, first identifiers of a first set of reference pointsmodeled on the three-dimensional model of the patient's dentition arereceived. The first set of reference points may correspond to a set ofanatomical points on the patient's dentition. In some implementations,the first set of reference points are provided by a technician or otherclinical professional who has identified specific anatomical portions ofthe 3D representation of interest to make photo-realistic and/orhumanistic. The first set of reference points may be chosen to minimize“uncanny valley” problems with machine representations of the patient'sdentition. In some embodiments, the reference points may include thegingival apex of one or more teeth, such as the anterior teeth. In someembodiments, the reference points may include the midpoint of theincisal edge of the teeth or the ends of the incisal edges of teeth. Insome embodiments, the reference points may be a cusp tip of one or moreteeth, such as the cusp tips of the canine teeth. In some embodiments,facial landmarks and contours, selected using neural networks, orotherwise, as descried herein, may be used as reference points. Examplesof reference points are further shown in FIGS. 7, 8, 9, 14 and 15 anddiscussed further herein.

At an operation 588, second identifiers of a second set of referencepoints modeled on the dentition of the image of the patient arereceived. The second set of reference points may correspond to the setof anatomical points on the patient's dentition. The second set ofreference may be provided by the technician or other clinicalprofessional.

At an operation 590, the image of the patient's dentition may beprojected into a three-dimensional space to create a projected 3D modelof the image of the patient's dentition. In various implementations,points at 2D coordinates of the image may be mapped into a 3D space sothat representations of 2D features in the 3D space are identified.

At an operation 592, the first set of reference points on the 3D modelof the patient's dentition may be aligned with the second set ofreference points on the projected model of the image of the patient'sdentition. In various implementations, the distance between one or morepairs of reference points or features is minimized. In some embodiments,the 3D bite model is rotated or translated about one or more of thethree perpendicular axes of 3D space. In some embodiments, the size ofthe teeth relative to the 2D image may be selected by matching thedistance and focal length of the rendering of the 3D bite model withfocal length and the distance between the imaging device and the patientused when capturing the 2D image of the patient.

At an operation 594, instructions to display a modified image of thepatient are provided. The modified image may represent the aligned firstand second sets of reference points. In some implementations, themodified image is a highly photo-realistic, humanistic image of thepatient's dentition. The modified image may include the estimatedresults of an intermediate or final stage of the orthodontic treatmentplan. The modified image may accommodate various perspectives, zooms,angles, etc. of the patient's dentition as informed through data modeledin the 3D model.

FIG. 6 depicts a two-dimensional image 600 of a patient, in accordancewith one or more embodiments herein. The 2D image 600 may be received orcaptured as described above with respect to the method 500. The image600 of the patient includes a 2D image of the representation of thepatient's face that may be created by and received from an imagingdevice, such as a camera. In some implementations, the 2D image may havebeen captured with a camera, uploaded with a phone or computer, and/oruploaded over a network (e.g., the Internet). The 2D image 600 of thepatient's face includes a mouth opening 610 and the patient's teeth andgingiva. The patient's mouth may be defined by the inner edges of theupper lip 620 and the lower lip 630. In particular, the outer perimeterof the mouth opening 610 may be defined by the lower edge of the upperlips 620, or a portion of the lower edge of the upper lips 620, and theupper edge of the lower lips 630, or a portion of the upper edge of thelower lips 630. FIG. 13D shows a process of finding lip edges andcontours.

FIG. 7 depicts a 3D bite model 700 of a patient's teeth within themouth. The 3D bite model may be formed as described above with referenceto FIGS. 1-4. As shown in FIG. 7, the 3D bite model includes 3D modelsof the patient's upper and lower jaws, including the teeth 710 and gums720 of the upper and lower jaws of the patient. In some embodiments, the3D bite model is a model of the patient's teeth in a pre-treatment orinitial position. In some embodiments, the 3D bite model depicts theteeth in the same relative positions as the relative positions of theteeth in the 2D image of the patient. For example, the 2D image of thepatient and scan of the patient's teeth may have been created beforetreatment starts, such as, for example, on the same day or within a timeperiod such that the position of the teeth in the 3D scan from which the3D bite model is built substantially matches the position of the teethat the time the 2D image of the patient is captured.

FIG. 8 depicts the selection of reference points 810 a, 810 b, 820 a,820 b on the 2D image of a patent. The selection of the reference points810 a, 810 b, 820 a, 820 b may be selected according to the processdescribed above with respect to FIGS. 5A and 5B and block 530. In someembodiments, the reference points 810 a, 820 a may be the gingival apexof one or more teeth, such as the anterior teeth. In some embodiments,the reference point 810 b may be the midpoint of the incisal edge of theteeth. In some embodiments, the reference point 820 b may be a cusp tipof one or more teeth, such as the cusp tips of the canine teeth.

The references points of the 2D image may correspond to correspondingreference points at the same location on the 3D bite model. FIG. 9depicts the selection of reference points 910 a, 910 b, 920 a, 920 b onthe 3D bite model of the patient's teeth. The selection of the referencepoints 910 a, 910 b, 920 a, 920 b may be selected according to theprocess described above with respect to FIGS. 5A and 5B and block 530.In some embodiments, the reference points 910 a, 920 a may be thegingival apex of one or more teeth, such as the anterior teeth. In someembodiments, the reference point 910 b may be the midpoint of theincisal edge of the teeth. In some embodiments, the reference point 920b may be a cusp tip of one or more teeth, such as the cusp tips of thecanine teeth.

FIGS. 10 and 11 depict the integration of the 3D bite model 1040 intothe 2D image 600 of the patient as part of the process to create acomposite image 1000. The integration of the 3D bite model 1040 into the2D image 600 depicted in FIGS. 10 and 11 may proceed as described abovewith reference to FIGS. 5A and 5B and block 540. In some embodiments,combining the 3D bite model 1040 with the 2D image 600 of the patientincludes determining a mouth opening 1010 in the 2D image. In someembodiments, the mouth opening 1010 is the shape of the inside edge ofthe patient's lips 1020, 1030 in the 2D image. In some embodiments, theportion of the 2D image within the mouth opening 1010 is deleted orotherwise removed from the 2D image 600 of the patient. In someembodiments, the 3D bite model 1040, or a 2D projection of the 3D bitemodel 1040 is placed or rendered behind the 2D image 600 such that the3D bite model 1040, or at least a portion of the 3D bite model 1040, isvisible through the mouth opening of the 2D image of the patient.

As shown in FIG. 11, the 3D bite model 1040 is manipulated andpositioned such that the teeth of the 3D bite model 1040 are in the sameposition and the same size as the teeth of the patient in the 2D image1000.

FIG. 11 depicts a close up of a composite image 1000 of the 3D bitemodel and the 2D image 600 of the patient with overlaid contours 1110 a,1110 b and reference points 810 a, 810 b, 820 a, 820 b, 910 a, 910 b,920 a, 920 b. As depicted in FIG. 11, two corresponding reference pointsmay be considered a pair of corresponding reference points. For example,a reference point 910 a at the left incisal gingival apex of the 3D bitemodel can correspond with the reference points 810 a at the left incisalgingival apex of the 2D image of the patient. As additional examples,reference points 810 b, 910 b are a corresponding pair of referencepoints at respective incisal edge midlines, reference points 820 a, 920a are a corresponding pair of reference points at a respective gingivalapex of the canine, and reference points 820 b, 920 b are corresponded apair of reference points at respective canine cusps.

The alignment of the 3D bite model with the 2D image may be performed asdescribed above with reference to FIGS. 5A and 5B and block 540 andelsewhere herein, for example as describe with reference to FIGS. 13A-F.The 3D bite model 1040 is manipulated and positioned such that the teethof the 3D bite model 1040 are in the same position and the same size asthe teeth of the patient in the 2D image 600. In some embodiments, the3D bite model 1040 is positioned in the 2D image 600 by aligning each ofthe reference points on the 3D bite model 1040 to each of thecorresponding reference points on the 2D image 600 of the patient. Insome embodiments, the reference points of the 3D bite model 1040 arealigned with the corresponding reference points on the 2D image 600 ofthe patient by minimizing the sum of the distance between eachcorresponding pair of reference points. In some embodiments, thereference points of the 3D bite model 1040 are aligned with thecorresponding reference points on the 2D image 600 of the patient byminimizing the sum of the squares of the distance between eachcorresponding pair of reference points.

In some embodiments, the 3D bite model 1040 and the 2D image 600 of thepatient may be aligned based on the alignment of tooth contours 1110 a,1110 b for the 3D bite model 1040 and the 2D image 600 of the patient.For example, the alignment may minimize the distance or square of thedistance between respective points that define the contours 1110 a, 1110b of the 3D bite model 1040 and the 2D image 600 of the patient.

In some embodiments, the 3D bite model may be aligned when the distancebetween all the reference points is less than a threshold, such as lessthan 0.05 mm, 0.1 mm, 0.2 mm, or 0.5 mm or wherein the sum of thesquares of the distances between the reference points is less than athreshold or minimized.

FIG. 12 depicts a composite image 1200 of the 3D bite model 1040 and the2D image of the patient with attachments 1210 placed on the patient'steeth. Attachments may be added to the 3D bite model during thetreatment planning process to aid in applying forces to the teeth.

FIG. 13 depicts a composite image 1300 of the 3D bite model 1040 withteeth in a final position and the 2D image of the patient. The 3D bitemodel 1040 and the 2D image of the patient may be aligned according toone or more embodiments described herein. Color and other featurematching may also be performed accordingly to one or more embodimentsdescribed herein.

FIG. 13A depicts a method 4600 for extracting mouth and teeth contoursfrom a patient's 2D image and aligning the image contours with contoursof a 3D model of the patients teeth.

At block 4610 the method 4600 determines the facial landmarks from afacial image. The facial image may be a two-dimensional facial image ofthe patient. The patient's facial landmarks within the image arecomputed using a machine learning algorithm, such as a convoluted neuralnetwork, to determine facial features such as the jawline, thecenterline of the face, the location of the eyes and eyebrows, thelocation of features of the nose, and also landmarks along the mouthopening and lips.

At block 4620 the shape and contours of the mouth opening are identifiedbased on the facial landmarks determined at block 4610. For example,each of the landmarks identified as being located on an inner lipcontour may be identified and contour lines may be drawn between each ofthe identified landmarks to determine the mouth opening. These faciallandmarks may be used to detect a mouth region and to conduct a courseplacing of the 3D model inside the mouth.

At block 4630 the contours within the mouth opening are extracted. Aconvolutional neural network may be used to extract the contours of thelips, gingiva, and teeth from within the mouth opening. In someembodiments holistic edge detection architecture of a convolutionalneural network is used to detect these and other contours describedherein.

At block 4640 the contours extracted from the facial image are matchedwith the contours of a three-dimensional model of the patient's teeth.In some embodiments the 3D model of the patient's teeth are in aninitial or pre-treatment position. For example the 3D model of thepatient's teeth may have been derived from a scan of the patient's teethtaken at or around the same time as the image of the patient's face.

FIGS. 13B and 13C depict some of the operations of the method 4600 forextracting mouth and teeth contours from a patient's two-dimensionalimage in more detail.

FIG. 13B depicts a facial image 4650 of a patient with facial landmarks4652 of a patient's face 4651 identified according to a machine learningalgorithm. The facial landmarks 4652 may include nasal centerlinelandmarks 4652A, facial outline landmarks 4652C, and inner lip landmarks4652D, among other landmarks such as landmarks that identify thepatients ears, eyebrows, eyes, features of the nose, chin, and otherfacial features. These facial landmarks can then be used to identifyaspects of the patient's face. For example, the inner lip landmarks maybe used to identify the mouth opening, within which the teeth andgingiva are located.

FIG. 13C shows the identification of the patient's lips, teeth, andgingiva. At panel 4660, the initial determination of the patient's lips4664, teeth 4662, and gingiva 4666 contours are identified by a machinelearning algorithm, such as a convoluted neural network.

At panel 4670, the initial tooth contours 4674 are extracted from themouth area. In some embodiments, the lips, gingiva, and other facialcontours identified in the facial image are removed from the image,resulting in an extraction of the tooth contours 4674 shown in panel4670. The tooth contours 4674 shown in panel 4670 have a brightness orother scale applied to them. For example, in a grey scale image of thecontours, each pixel may be assigned a value between 0 and 255, whichmay indicate a confidence that the pixel is a contour or may indicatethe magnitude of the contour at that location.

At panel 4680, the tooth contours, and in particular, the pixels thatdenote the contours undergo binarization to change the pixels from ascale of, for example, 0 to 255, to a binary scale, for example, of 0 or1, creating binarized tooth contours 4682. In the binarization process,the value of each pixel is compared to a threshold value. If the valueof the pixel is greater than the threshold, then it may be assigned anew first value, such as a value of 1 and if the pixel is less than thethreshold, then it may be assigned a new second value, such as a valueof 0, for example.

At panel 4690, the binarized tooth contours 4682 from panel 4680 arethinned, whereby the contour's thickness is reduced to, for example, asingle pixel in width, forming a thinned tooth contour 4692. The widthof a contour for thinning may be measured as the shortest distance froma contour pixel adjacent to a non-contour pixel on a first side of acontour, to a contour pixel adjacent a non-contour pixel on a secondside of the contour. The single pixel representing the thinned contourat a particular location may be located at the midpoint of the widthbetween the pixel at the first side and the pixel at the second side.

After thinning the binarized tooth contours 4682, the thinned contour4692 may be a single width contour at a location that corresponds to amidpoint of the binarized contour 4682.

After forming the thinned contour 4682 of the teeth, the thinnedcontours 4682 may be combined with lip contours, described below, andthe facial image, as shown, for example, in FIG. 13E, discussed below.

FIG. 13D shows a process of identifying a patient's lip contours andmouth opening. At panel 4710 an image of a patient's face, and inparticular the mouth and the region near the mouth, is shown. The lipcontours and mouth opening are determined based on such an image.

The initial determination of the patient's lips may be based on faciallandmarks, such as the lip landmarks determined according to a machinelearning algorithm. For example, the facial landmarks may include innerlip landmarks 4652D shown in FIG. 13D.

In some embodiments, the 3D model of the patient's teeth is initiallycoarsely placed within the mouth region. The initial placement is basedon an alignment of the midline of the 3D model with a midline of thefacial image determined based on the midline facial landmarks, such asthe nasal midline landmarks 4652 a of FIG. 13B. The initial scale of the3D model may be determined by matching the scale of a distance betweentips of opposite teeth, such as, by matching the distance between theupper canines in the 3D model with the distance between the tips of theupper canines in the facial image or matching the size of one or moreteeth in the 3D model with the site of one or more corresponding teethin the facial image.

Panel 4720 shows the initial determination of the patient's lip contours4722, as identified by a machine learning algorithm, such as aconvoluted neural network. These initial lip contours 4722 are extractedfrom the mouth area. In some embodiments, the tooth, gingiva, and otherfacial contours identified in the facial image are removed from theimage, resulting in an extraction of the lip contours 4724 shown inpanel 4720. In some embodiments, other facial contours 4722 may bepresent at this stage in the process.

As with the tooth contours 4674 shown in panel 4670, the lip contours4722 have a brightness or other scale applied to them. For example, in agrey scale image of the contours, each pixel may be assigned a valuebetween 0 and 255, which may indicate a confidence that the pixel is acontour or may indicate the magnitude of the contour at that location.

At panel 4730, the lip contours, and in particular, the pixels thatdenote the contours undergo binarization to change the pixels from ascale of, for example, 0 to 255, to a binary scale of, for example, 0 or1, creating binarized lip contours 4732. The binarization process issimilar to that described above with respect to FIG. 13C.

At panel 4740, the binarized lip contours 4732 from panel 4730 arethinned, whereby the contour's thickness is reduced to, for example, asingle pixel in width, and forming a thinned lip contour 4742. Asdescribed above with respect to FIG. 13B, the width of a contour forthinning may be measured as the shortest distance from a contour pixeladjacent to a non-contour pixel on a first side of a contour, to acontour pixel adjacent a non-contour pixel on a second side of thecontour. The single pixel representing the thinned contour at aparticular location may be located at the midpoint of the width betweenthe pixel at the first side and the pixel at the second side.

After thinning the binarized lip contours 4732, the thinned contour 4742may be a single pixel width contour at a location that corresponds tothe midpoint of the binarized contour 4732. The thinned contour imagemay still include other facial contours 4744 that are not part of thelip contours 4742. Accordingly, the largest connected component of theimage may be determined to be the lip contours. This is possible, inpart because, at the onset of the process, the 2D facial image includedthe mouth region and excluded or cropped out many, most, or all of theother facial features, such as one or more of the chin, jaw, eyes, ears,and most, or all of the nose.

Panel 4750 shows the extracted single pixel width lip contour 7452 alongwith four basic points of the lip contours. The four basic points maycorrespond to the left-most point 4757 of the lip contour 4752, themiddle point of upper lip contour 4755 of the lip contour 4752, theright-most point 4754 of the lip contour 4752, and the middle point oflower lip contour 4756 of the lip contour 4752.

In some embodiments, shortest paths between the left-most point 4754 andthe middle points 4755, 4756 and shortest paths between the right-mostpoint 4754 and the middle points 4755, 4756 are determined, such paths4762 are depicted in panel 4760.

In some embodiments, the shortest paths are refined based on the lipcontours. For example, an iteratively built spline may be formed basedon one or more of the shortest paths and the lip contour 4752. Forexample, an initial point of the spline may be located on the leftcorner of the mouth contour, which may correspond to the left-most point4757. From this initial point a spline is built rightward along thelower lip towards the middle point of the lower lip contour 4756. When,advancement of the spline diverges from lower lip, as determined forexample by maximizing R-square or another method, advancement stops anda new point is placed. The divergence may also be determined based on asum of the square of the distance between the spline and the lipcontours at points along the spline, or another method. The process thenrepeats until one or more splines are built between each of the fourbasic points. In some embodiments, the splines form a closed perimeterthat corresponds to the mouth opening of the facial image.

After the spline or splines 4772 are formed, the splines 4772 may becombined with the facial image, as shown, for example, in panel 4770.

FIG. 13E shows the facial image with the contours 4782 of the 3D toothmodel and lip contours 4784 overlaid thereon. To match the 3D toothmodel tooth contours to the facial image, an expectation-maximization(EM) algorithm is used. First, the 3D tooth model contours are projectedin an image plane. In some embodiments, the image plane is determinedbased on the distance between the imager or camera used to take thefacial image and the focal length of the lens used to take the facialimage. Then for the expectation step of the EM algorithm, for each pixelon the tooth contours from the facial image is matched to a similarpixel found on the 3D tooth model. On the maximization step of the EMalgorithm the jaws or tooth arches of the 3D model are adjusted in oneor more of translation and rotation in one or more of three orthogonaldirections to minimize the total discrepancies between the pixels fromthe facial contours with pixels from the 3D model contours. In someembodiments, camera parameters may also be adjusted minimize totaldiscrepancy. The total discrepancy may be determined based on a sum ofleast squares method or another method. In some embodiments, a pin-holecamera projection may provide an accurate projection of the 3D toothmodel contours.

FIG. 13E shows the aligned lip contours 4784 and 3D model contours 4782within the facial image 4780. The alignment accounts for the portion ofteeth contours 4782 a that are visible though the mouth opening definedby the lip contours 4784 and the portions of teeth contours 4782 b thatare not visible though the mouth opening. When rendering a composite 2Dand 3D image (for example, as shown in FIGS. 12 and 13), the visibleportions of the 3D tooth model may be rendered while the nonvisibleportions of the 3D tooth model may not be rendered. For example, theteeth or other portions of the 3D tooth model, also referred to as abite model, obscured by the patients lips, may not be rendered. Theprocess of forming the composite 2D and 3D image may be performed at oneor more of blocks 540 through 560 of FIGS. 5A and 5B.

FIGS. 23A-23D show a method 5000 of determining facial landmarks to aidin integrating a 3D tooth model into a facial image of a patient. Atblock 5010 facial landmarks from a facial image are determined. In someembodiments, the facial landmarks are determined based on a neuralnetwork that may have been trained on human faces. FIG. 23B shows afacial image 5060 with identified landmarks. The landmarks includeeyebrow landmarks 5062 that identify the upper edge of a person'seyebrow, eye landmarks 5066 that identify the corners and lids of aperson's eye, nasal landmarks 5068 that identify features of a person'snose, mouth landmarks 5061 that identify features of a person's lips andmouth, and facial outline landmarks 5064 that identity the outline of aperson's face, such as the chin and jaw lines.

At block 5020 the facial midline of the facial image 5060 is determinedbased on the locations of the facial landmarks. In some embodiments, asubset of facial landmarks are used to determine the facial midline 5080of the face in the facial image 5060. For example, in some embodiments,four central landmarks 5072 are used to determine the facial midline.These central landmarks 5072 are typically aligned along the facialmidline and include a landmark 5072 a along the nasal ridge between theeyes, a landmark 5072 b at the tip of the nose, landmark 5072 c and thecenter of the chin, and landmarks 5072 d, 5072 e at the center of theouter lower and upper lips, respectively.

In some embodiments, the average of symmetrical facial landmarks may beused to determine the facial midline. Symmetrical facial landmarks arepairs of landmarks that identify the same feature on each side of theface. For example, the pair of central eyebrow landmarks 5074 include aleft central eyebrow landmark and a right central eyebrow landmark.Similarly, the other eyebrow landmarks may be used, such as the inner5075 and outer 5076 canthus, the ala 5073.

These landmarks are generally equidistant from the facial midline.Accordingly, an average of the location of a pair of symmetrical faciallandmarks is determined based on the location of each symmetricallandmark of the pair. This average location may be used to determine amidline point. The midline points from each pair of symmetrical pointsis then used to determine the facial midline 5080.

In some embodiments, both symmetrical facial landmarks and faciallandmarks 5072 that are generally along the facial midline may be usedto determine the facial midline 5080.

When determining the facial midline using either the facial landmarks5072, the symmetrical landmarks, or a combination of both, the facialmidline may be determined at a straight line that maximizes theR-squared of the distance between the facial midline 5080 and therespective landmarks, such as the landmarks 5072 and the averagelocation of each pair of symmetrical landmarks. In some embodiments, theone or more landmarks may be weighted, such that their location has agreater or lesser effect on the position of the midline. For example,facial landmarks 5072, which may be referred to as midline landmarks maybe weighted greater than symmetrical landmarks.

At block 5030 (see FIG. 23A) a facial plane is determined. A facialplane may be formed based on the facial midline 5080 and the positionand focal length of the imager used to capture the facial image. Thefacial plane may be a 2D plane extending though the facial midline 5080and may be parallel to the sagittal plane of the patient. A facial planemay be used to align a rendering of a 3D model of the patient's teeth,for example in initial, final, or intermediate positions, with thefacial image of the patient.

FIG. 23D shows a 3D model of a patient's teeth 5090 and a facial plane5092. The 3D model includes a dental midline 5094. The dental midlinemay be a line though the midpoint of the patient's arch, for example,the dental midline 5094 may extended between the upper central incisors.In some embodiments, the dental midline 5094 extends in a directionperpendicular to the occlusal plane of the patient. The dental midline5094 may also be determined as described herein with reference to FIG.18. To align the 3D model 5090 with a facial image of a patient, thefacial plane 5092 and the dental midline 5094 are aligned by rotatingand translating one or more of the 3D model 5090 and the facial plane5092 until the dental midline 5094 is in the facial plane 5092. Therotations and/or translations of the facial plane 5092 are also carriedout on the facial image. At block 5040 (see FIG. 23A), rotations of thefacial image or photo are carried out. In some embodiments, the 3D modelor the facial image may be rotated and translated.

In some embodiments, no rotations or translations of the facial imageare carried out. Instead, the 3D model is rotated or translated untilthe dental midline 5094 is in the facial plane 5092.

After alignment of the facial image and the 3D model, such as describedabove by aligning the dental midline with the facial plane, the 3D modelis inserted into the facial image or image of the patient.

FIG. 13F shows a process of morphing a rendering from a pre-treatmenttooth arrangement shown in panel 4791 to a post-treatment or plannedfinal tooth arrangement shown in panel 4793. The morphing algorithm mayuse a facial image 4790, the lip contours 4792, and masks of the teethrendered in initial tooth positions 4794 and final tooth positions 4796as inputs. The initial 2D and 3D composite image shown in panel 4791 maybe rendered using the initial 2D facial image and the 3D model of thepatient's teeth rendered in the initial position and the final compositeimage shown in panel 4793 may be rendered using the initial 2D facialimage and the 3D model of the patient's teeth rendered in the finalposition.

To morph between the initial and final tooth positions, a segmentationmask may be built for both the rendering of the teeth in the initialposition and the rendering of the teeth in the final positions. Suchsegmentation masks may include corresponding triangulation such thateach triangle in the rendering of the first position corresponds to atriangle in the final position. The corresponding triangles may, andlikely do, have different shapes in the initial and final renderings.The resulting morph between the teeth in the initial position and theteeth in the final position is a frame-by-frame translation of thetriangles shaped according to the initial position to the trianglesshaped according to the final position with corresponding color andshading translations.

In some embodiments, the incisal edges of teeth may be used to align a3D model of a patient's teeth with a 2D facial image of a patient. Forexample, FIG. 14 depicts an embodiment of the selection or determinationof reference points along the incisal edge or cusps of the teeth on the2D image of a patent. The selection of the reference points 1410, 1420,1430, 1440 may be made according to the process described above withrespect to FIGS. 5A and 5B and block 530. In some embodiments, thereference points 1410, 1420 may be the midpoint of the incisal edge ofthe teeth. In some embodiments, the reference points 1430, 1440 may be acusp tip of one or more teeth, such as the cusp tips of the canineteeth.

The references points of the 2D image may correspond to similarreference points at the same location on the 3D bite model. FIG. 15depicts the selection of reference points 1510, 1520, 1530, 1540 on the3D bite model of the patient's teeth. The selection of the referencepoints 1510, 1520, 1530, 1540 may be selected according to the processdescribed above with respect to FIGS. 5A and 5B and block 530. In someembodiments, the reference points 1510, 1520 may be the midpoint of theincisal edge of the teeth. In some embodiments, the reference point1530, 1540 may be a cusp tip of one or more teeth, such as the cusp tipsof the canine teeth.

The integration of the 3D bite model 1040 into the 2D image may proceedas described above with reference to FIGS. 5A and 5B and block 540. Insome embodiments, combining the 3D bite model 1040 with the 2D image 600of the patient includes determining a mouth opening 1010 in the 2Dimage. In some embodiments, the mouth opening 1010 is the shape of theinside edge of the patient's lips 1020, 1030 in the 2D image. In someembodiments, the portion of the 2D image within the mouth opening 1010is deleted or otherwise removed from the 2D image 600 of the patient. Insome embodiments, the 3D bite model 1040, or a 2D projection of the 3Dbite model 1040 is placed or rendered behind the 2D image 600 such thatthe 3D bite model 1040, or at least a portion of the 3D bite model 1040,is visible through the mouth opening of the 2D image of the patient. The3D bite model 1040 is manipulated and positioned such that the teeth ofthe 3D bite model 1040 are in the same position and the same size as theteeth of the patient in the 2D image 600.

Two corresponding reference points may be considered a pair ofcorresponding reference points. For example, a reference point 1510 atthe left incisal edge of the 3D bite model can correspond with thereference points 1410 at the left incisal edge of the 2D image of thepatient. As additional examples, reference points 1430, 1530 are acorresponding pair of reference points at respective canine cusp tips.

The alignment of the 3D bite model with the 2D image may be performed asdescribed above with reference to FIGS. 5A and 5B and block 540. The 3Dbite model 1040 is manipulated and positioned such that the teeth of the3D bite model 1040 are in the same position and the same size as theteeth of the patient in the 2D image 600. In some embodiments, the 3Dbite model 1040 is positioned in the 2D image 600 by aligning each ofthe reference points on the 3D bite model 1040 to each of thecorresponding reference points on the 2D image 600 of the patient. Insome embodiments, the reference points of the 3D bite model 1040 arealigned with the corresponding reference points on the 2D image 600 ofthe patient by minimizing the sum of the in distance between eachcorresponding pair of reference points. In some embodiments, thereference points of the 3D bite model 1040 are aligned with thecorresponding reference points on the 2D image 600 of the patient byminimizing the sum of the squares of the distance between eachcorresponding pair of reference points. In some embodiments, the 3D bitemodel may be aligned when the distance between all the reference pointsis less than a threshold, such as less than 0.05 mm, 0.1 mm, 0.2 mm, or0.5 mm or wherein the sum of the squares of the distances between thereference points is less than a threshold or minimized.

Virtually Representing a Treatment Outcome Using Automated Detection ofFacial and Dental Reference Objects

In some implementations, one or more modules of the data processingsystem 400 may be configured to virtually represent an orthodontictreatment outcome by detecting and comparing facial and dental referenceobjects against one another. A “reference object,” as used in thiscontext, may refer to one or more physical and/or anatomical features ofa patient that can be used to infer the placement of portions of digital3D models within a 2D image. A “reference object” may be compriseelements of a patient's face or a patient's teeth/dentition. Examples ofa facial reference object include: a facial midline along the sagittalplane of a patient's face; a location of the inferior boarder of theupper lip at a patient's facial midline; a location of the superiorboarder of the lower lip at the facial midline; etc. Examples of adental reference object include: a dental midline; an incisal edgeposition of one or more teeth; a gingival zenith of a central incisor;etc.

In various implementations, the processor(s) 402 gather from the storagesubsystem(s) 406 a 3D model that models an initial position of apatient's teeth. The 3D model may include virtual representations ofeach of the patient's teeth and may depict translations and/or rotationsof the patient's teeth along six degrees of freedom. The 3D model mayinclude virtual representations of the patient's teeth at an initialpre-orthodontic treatment stage as well as along intermediate and/orfinal orthodontic treatment stages.

The camera 425 may be configured to capture an image of the patient'sface. In some implementations, the camera 425 is located remotely to theother modules of the data processing system 400. As an example, thecamera 425 may comprise a dedicated camera incorporated into anintraoral scanner, a cellphone camera, a network-coupled camera, aradiograph, scans related to PVS impressions, etc.

In some implementations, the processor(s) 402 may executecomputer-implemented instructions stored on the storage subsystem 406 toselect facial reference objects on images of the patient's face. Asnoted herein, the facial reference object may include one or more of: afacial midline along the sagittal plane of a patient's face; a locationof the inferior boarder of the upper lip at a patient's facial midline;a location of the superior boarder of the lower lip at the facialmidline; etc. Further, the processor(s) 402 may executecomputer-implemented instructions stored on the storage subsystem 406 toselect dental reference objects on the patient's teeth. The dentalreference objects may be gathered from the 3D model of the patient'sdentition. The processor(s) 402 may further execute computer-implementedinstructions stored on the storage subsystem 406 to assign locations tothe facial reference object(s) and/or the dental reference object(s). Invarious implementations, the processor(s) 402 may executecomputer-implemented instructions stored on the storage subsystem 406 tocompare locations of facial reference objects with locations of dentalreference objects.

Advantageously, the processor(s) 402 may execute computer-implementedinstructions stored on the storage subsystem 406 to modify interimpositions of teeth on the 3D model using comparisons between locationsof facial reference objects and locations of dental reference objects.In some implementations, modifications may be recommended and/orimplemented. Depending on the implementation, the processor(s) 402 mayonly make modifications if a specified threshold value is reached,exceeded, etc.

As an example, the processor(s) 402 may execute computer-implementedinstructions stored on the storage subsystem 406 to modify interimpositions of teeth on the 3D model using comparisons between a facialmidline and a dental midline; to the extent the dental midline does notmatch up with the facial midline, the dental midline of the 3D model maybe modified. As another example, the processor(s) 402 may executecomputer-implemented instructions stored on the storage subsystem 406 tomodify interim positions of teeth on the 3D model using comparisonsbetween an inferior boarder of the upper lip at the facial midline andincisal edge position; to the extent the incisal edge position does notmatch up with the inferior boarder of the upper lip at the facialmidline, the incisal edge position of the 3D model may be modified.

As yet another example, the processor(s) 402 may executecomputer-implemented instructions stored on the storage subsystem 406 tomodify interim positions of teeth on the 3D model using comparisonsbetween an superior boarder of the lower lip at the facial midline andincisal edge position; to the extent the incisal edge position does notmatch up with the superior boarder of the upper lip at the facialmidline, the incisal edge position of the 3D model may be modified. Asyet another example, the processor(s) 402 may executecomputer-implemented instructions stored on the storage subsystem 406 tomodify interim positions of teeth on the 3D model using comparisonsbetween an inferior boarder of the lower lip at the facial midline andgingival zenith of a central incisor; to the extent the gingival zenithof a central incisor does not match up with the inferior boarder of theupper lip at the facial midline, the gingival zenith of a centralincisor of the 3D model may be modified.

In some implementations, the processor(s) 402 may executecomputer-implemented instructions stored on the storage subsystem 406 toidentify a facial type of a patient. A “facial type,” as used herein,may include a class of face shapes and/or characteristics from which toinfer location of dentition after application of an orthodontictreatment plan. The facial type is determined based on a distancebetween the patient's glabella and chin and the distance between thepatient's right and left cheekbone prominences. The processor(s) 402 mayexecute computer-implemented instructions stored on the storagesubsystem 406 to modify widths of specific teeth, such as centralincisors, lateral incisors, and canines in an interim final positionrepresented on the 3D model. In various implementations, theprocessor(s) may execute computer-implemented instructions stored on thestorage subsystem 406 to accommodate the size(s), shape(s), location(s),and relationships of restorative objects, such as crowns, veneers, etc.

FIG. 16A depicts an embodiment of a method 1600 for determining a finalposition of a patient's teeth based, in part, on facial features, toothproportions and positions, and gum positions. The method 1600 may bepart of the process of developing a treatment plan and set of alignersfor a patient as shown and described above, in particular with referenceto FIGS. 1-4. In some embodiments, method 1600 takes place after theafter an initial treatment planning process as described with referenceto FIGS. 1-4.

At block 1602 the process of determining the facial midline andinterpupillary line is conducted. FIG. 17 illustrates one embodiment ofselecting, identifying or otherwise determining the facial midline andinterpupillary line of a patient from the 2D image of the patient.Another method is shown and described with respect to FIGS. 23A-D. Insome embodiments, the patient is imaged with a social smile facialexpression. In some embodiments, a point of the subnasion 1730 and theglabella 1710 are selected or otherwise determined. A facial midline1740 is drawn from or through the point of the glabella 1710 to orthrough the point of the subnasion 1730. In some embodiments, the pointof subnasion 1730 is defined, and it remains static. In someembodiments, the image of the patient or the face of the patient in theimage may be in a non-vertical orientation. In such embodiments, theimage of the patient may be rotated until the facial midline 1740 is ina vertical orientation. In some embodiments, the image or face of thepatient in the image may be rotated about either the point of thesubnasion 1730 or the glabella 1710. In some embodiments aninterpupillary line 1760 may be drawn through or between the points ofthe eye pupils 1720 a and 1720 b.

At block 1604 of FIG. 16A, the dental midline from a 2D image of apatient with a social smile is selected or otherwise determined. Asshown in FIG. 18, the dental midline 1810 may be selected or otherwisedetermined based on the location of the papilla 1820, the incisalembrasure 1830 of the anterior central incisors 1840, or the incisalembrasure between the lower incisors. The papilla 1820 is a smallsounded protuberance on the gingiva and the incisal embrasures 1830 arethe v-shaped valleys between adjacent teeth, such as the two anteriorcentral incisors 1840. The dental midline 1810 may be defined as a linethrough the upper incisal embrasure 1830 of the anterior centralincisors 1840, or the lower incisal embrasure between the lower incisorsand parallel to the facial midline 1740. In some embodiments, a distancebetween the facial midline and the dental midline is determined. In someembodiments, if the distance between the facial midline and the dentalmidline is greater than a threshold value, such as 0.05 mm, 1 mm, 2 mm,3 mm, 4 mm, or 5 mm, the final position of the teeth in the treatmentplan may be determined such that that the dental midline is moved suchthat it is less than the threshold. In some embodiments, the thresholdfor movement is different than the threshold for the final position. Forexample, in some embodiments, the threshold distance below which thetreatment planning does not adjust the dental midline is 2 mm, but oncethe treatment planning determines that the dental midline is moved, thenthe dental midline is shifted to below a second threshold value, such as1 mm or 0.5 mm or until the facial midline and the dental midline arecoincident.

At block 1606 the position of the incisal edge of the upper centralincisors 1960 from a 2D image of a patient with a repose facialexpression is selected or otherwise determined. As shown in FIG. 19, ahorizontal line perpendicular to the facial midline 1740 at thesubnasion 1743 may be generated or otherwise formed. A lineperpendicular to the facial midline 1740 at the inferior border of theupper lip 1940 (the lower edge of the upper lip) may be generated orotherwise formed. A line perpendicular to the facial midline 1740 at themost inferior incisal edge point of the upper central incisors 1950 (thelowest edge point of the incisors in the upper dental arch) may begenerated or otherwise formed. A line can be drawn perpendicular to thefacial midline 1740 at a target position of the inferior incisal edgepoint of the upper central incisors 1960. The target position may beselected or otherwise determined based on facial aesthetics. An inferiorposition is the direction towards the feet, while a superior position isthe direction towards the head.

Once these lines are generated, the distances between the lines may bemeasured to determine the spatial difference between the existing andthe target positions. In some embodiments, the distance may be measuredrelative to the subnasion 1730 and the inferior incisal edge point ofthe central incisors 1950 to determine the initial position of theincisal edge of the teeth. In some embodiments, the distance may bemeasured between the subnasion 1730 and the target position of theinferior incisal edge point of the upper central incisors 1960, and thisdistance may be used for determining the target position of the incisaledge of the teeth. In some cases, the distances may be measured withreference to the inferior border of the upper lip 1940, rather than thesubnasion 1730. In some embodiments, if the initial position of theincisal edge is outside of a target range, then the incisors may bemoved as part of the treatment planning process to the final targetposition such that the incisal edge is within a target distance range.In some embodiments, the target distance range between the inferiorborder of the upper lip 1940 and the target position 1960 of the incisaledges 1950 may be between 3 and 4 millimeters.

In some embodiments, at block 1606 the position of the incisal edge ofthe upper central incisors 1960 from a 2D image of a patient with asocial smile is determined. As shown in FIG. 20, a horizontal line maybe generated or otherwise formed perpendicular to the facial midline1740 through the subnasion 1730. A line may be drawn perpendicular tothe facial midline 1740 at the most inferior incisal edge point of theupper central incisors 1950. A line may be drawn perpendicular to thefacial midline 1740 at a target position 1960 of the inferior incisaledge point of the upper central incisors. The point of the superiorborder of the lower lip that intersects the facial midline 2010 may alsobe identified.

The distances between the lines may be measured. In some embodiments,the distances are measured to quantify the spatial difference betweenthe existing and the target positions. In some embodiments, thedistances may be measured between the subnasion 1730 and the inferiorincisal edge point of the central incisors 1950, to determine an initialdistance. In some embodiments, the distance may be measured between thesubnasion 1730 and the target position 1960 of the inferior incisal edgepoint of the upper central incisors, and this distance may be used fordetermining the target position of the teeth as part of the treatmentplanning process. In order to design an aesthetically pleasing smile,the distance between the target position of the inferior incisal edgepoint of the upper central incisors 1960 and the superior border of thelower lip intersecting the facial midline 2010 may be less than or equalto 1 millimeter. In some embodiments, this distance may be less than orequal to 2 mm, for example, when the patient's lower lip is a dynamic orV-shaped lip in a social smile expression. In the treatment planningprocess, the teeth may be moved such that in their final positions, theyare these threshold distances.

At block 1608 the gingival line reference from a 2D image of a patientwith a social smile 2120 is determined, for example, as shown in FIG.21. The location of the gingival zeniths 2140 may be selected orotherwise determined. The inferior border of the upper 1940 can also bemeasured. The distance between the gingival zenith 2140 and the inferiorborder of the upper lip 1940 may be determined based on a compositeimage of the 3D bite model and the 2D image of the patient or from the2D image of the patient without the 3D bite model. In some embodiments,the threshold distance between the gingival zenith and the inferiorborder of the upper lip, below which the gingival zenith may be alteredduring treatment planning may be 3 millimeters In some embodiments, thetarget range for the final distance after treatment is between −1millimeters (the gingival zenith is above the inferior border of theupper lip and hidden) to 2 millimeters. The central incisor height (CIH)may be measured from the gingival zenith to the edge of the tooth. Anaesthetic CIH can be less than or equal to 12 millimeters. In someembodiments, the treatment plan moves the teeth such that the CIH isless than 12 millimeters.

As described herein, an initial target final orthodontic position of thepatient's teeth may be determined at block 1610 of FIG. 16A. In someembodiments, block 1610 may also include the generation of a treatmentplan based on the initial orthodontic target position.

At block 1612 of FIG. 16A a face type of the patient may be determinedas shown in FIGS. 22 and 23. As shown in FIG. 22, reference points andthe distances between pairs of reference points may be selected,measured, or otherwise determined. The location of the glabella 1710 maybe determined at the intersection of the facial midline 1740. A point onthe chin 2240 at the intersection of the facial midline 1740 may also bedetermined. The distance between point on the chin 2240 and the point onthe glabella 1710 are used in conjunction with the distances betweenrespective pairs of the other points to determine the shape of thepatient's face.

A respective pair of points, one on each of the cheekbones 2210 a, 2210b and the distance between them can be determined. A respective pair ofpoints, one on each of the mandibular angles 2230 a, 2230 b and thedistance between them can be determined. A respective pair of points,one on each of the temples 2220 a, 2230 b and the distance between themcan be determined. Once these respective pairs of points and thedistances between them are measured or otherwise determined, thedistances between the points may aid in determining the spatial facialprofile and to categorize the facial type as shown in FIG. 23.

FIG. 23 shows three face types, a short face type 2310, and average facetype 2320, and a tall face type 2330. In a short-type face 2310, thedistance between glabella 1710 and the chin 2240 is similar to or equalto the distance between the cheekbones 2210 a and 2210 b and similar orequal to the distance between the mandibular angles 2230 a and 2230 b,for example, the distances are within 10% of each other.

In an average-type face 2320, the distance between the glabella 1710 andthe chin 2240 is much greater than the distance between the cheekbones2210 a and 2210 b, for example between 15% and 20%; the distance betweenthe cheekbones 2210 a and 2210 b is similar to or equal to the distancebetween the mandibular angles 2230 a and 2230 b; and the distancebetween the temples superior to the ears 2220 a and 2220 b is greaterthan the distance between the mandibular angles 2230 a and 2230 b, forexample, between 10% and 15%.

In a tall-type face 2330, the distance between the glabella 1710 and thechin 2240 is much greater than the distance between the cheekbones 2210a and 2210 b, for example greater than 20%; the distance between theglabella 1710 and the chin 2240 is much greater than the distancebetween the mandibular angles 2230 a and 2230 b, for example greaterthan 20%; and the distance between the glabella 1710 and the chin 2240is much greater than the distance between the temples superior to theears 2220 a and 2220 b, for example greater than 20%, while thedistances between the temples superior to the ears 2220 a, 2220 b, thedistance between the mandibular angle 2230 a, 2230 b, and the distancebetween the cheekbone prominence 2210 a, 2210 b, are all similar, forexample, within 10%.

At block 1614 the inter-canine width (ICW) is determined, for example,as shown in FIG. 24. The line that demarcates the inter-canine width2420 a and 2420 b may be selected or otherwise determined by verticallines through the most distal points on the buccal surfaces of the uppercanines 2410 a and 2410 b. These ICW lines 2420 a and 2420 b aredetermined based on the teeth being in the orthodontic final position ofthe 3D bite model, as shown in FIG. 24. These ICW lines 2420 a and 2420b are parallel to the dental midline 1810 and intersect the line drawnbetween the upper canines 2410 a and 2410 b.

At block 1616 the proportions of the upper anterior tooth widths may bedetermined. In some embodiments, the proportions of the upper anteriortooth widths may be determined based on the Recurring Esthetic Dental(RED) proportion or the upper anterior tooth width. Under REDproportions, the successive width proportion when viewed from the facialaspect should remain constant from the midline toward the posterior forthe six anterior teeth between and including the canines. This propertyoffers great flexibility to match tooth properties with facialproportions. Table 1 shows the RED proportions and anterior total widthsbased on the inter-canine-width (ICW) of the patient and the patient'sface type. During the treatment planning process the teeth may be movedand/or restorative objections may be used such that at the end oftreatment, the patient's teeth are within the RED proportions.

TABLE 1 Calculating RED Proportion & Anterior Total Widths fromInter-Canine Width (ICW) and Face Type Anterior Tooth Widths Desired REDProportion Central Incisor Lateral Incisor Canine Face Type RED % Width(CIW) Width (LIW) Width (CW) Tall 66% RED ICW/4.2 CIW/0.66 LIW/0.66Average 70% RED ICW/4.4 CIW/0.7  LIW/0.7  Short 75% RED ICW/4.6 CIW/0.75LIW/0.75

At block 1618 the central incisor proportion (CIP), which can bedetermined from the ratio of the central incisor width (CIW) or thewidth 2520 of the central incisor 2500 and the central incisor height(CIH) or the height of the central incisor 2500, is determined. In ashort-type face 2530, the CIP is less than or equal to 85%. In anaverage-type face 2540, the CIP is 78%. In a tall-type face 2450, theCIP is greater than or equal to 70%.

At block 1619 the real gingival line is determined, as shown in FIG. 26.The location of the final position of gingival zeniths 2140 can bedetermined. The inferior border of the upper 1940 can also bedetermined. The distance between the gingival zenith 2140 and theinferior border of the upper lip 1940 may also be determined. Thecalculated distances can be compared to acceptable gingival distances,as discussed above, and the final orthodontic position of the patient'steeth may be adjusted such that the distance is within a target range orbelow a target value.

At block 1620 restorations from a tooth library are placed on the 3Dbite model. FIG. 27 shows restorations from the tooth library 2710 thatcan be placed on the 3D bite model for creating the final 3D bite model.The initial reference points for the placement process can include, butare not limited to, the incisal edge 2720, the gingival line 2710, theline that demarcates the inter-canine width 2420 a and 2420 b at themost distal points on the buccal surfaces of the upper canines, REDpercentages, and central incisor proportions. In some embodiments, the3D tooth models from the tooth library are idealized or generic shapesof teeth. The shapes of these tooth models may be applied to the 3D bitemodel of the patient to aid in determining the shape of a restorativeobject to be used on the patient's teeth.

At block 1622 the smile arc of the patient's teeth in the finalorthodontic position with the restorative shapes is determined. As shownin FIG. 28, the following reference points may be determined: thecentral incisor edge 2810 a and 2810 b, the canine cusp 2820 a and 2820b, and the lip commissures 2830 a, 2830 b. The teeth from the toothlibrary may be modified such that the tooth surface shapes of the 3Dbite model 2840 match or more closely match the target reference pointsfor the central incisor edge 2810 a and 2810 b, the canine cusp 2820 aand 2820 b, and the lip commissures 2830 a, 2830 b. Then adjustments aremade on the canine tooth such that the length of the central incisor issimilar to or equal to the length of the canine. Reference points on thecentral incisor edge 2810 a and 2810 b and the canine cusp 2820 a and2820 b can be adjusted via the canine. The arc of the smile can becreated with a line intersecting the lip commissure 2830 a, the caninecusp 2820 a, the central incisor 2810 a, the central incisor 2810 b, thecanine cusp 2820 b, and the lip commissure 2830 b.

At block 1624 tooth inclination may be determined and modified. As shownin FIG. 29, reference lines for tooth inclination may be formed as linesextending between or through the gingival zenith 2910 and a center pointincisal edge points 2920 or tooth cusps. These reference lines may beused to determine the angle of the tooth to with respect to the dentalmidline. In some embodiments, the shape or position of the final targetpositions of the teeth may be adjusted such that the angles of eachreference line with the dental midline increases for each tooth awayfrom the dental midline. For example, the angle of the central incisorline is less than the angle of the reference line for the lateralincisor, which is less than the angle of the canine.

At block 1626 a final or revised final orthodontic position of the teethis determined and a treatment plan may be regenerated. In someembodiments, the positions of the teeth may be generated and displayedas shown in FIG. 30, which illustrates the 3D bite model 3020 integratedwith a facial image of the patient 3010. At block 1628, the finalorthodontic and restorative positions of the teeth are evaluated andfeedback may be provided for further revisions to the final position andthe treatment plan. FIG. 31 illustrates the 3D bite model 3120 afterimplementation of the treatment plan. The treatment plan can include anevaluation of the restorative overlay, a tooth mass reductionassessment, and an assessment of restorative option versus orthodonticsolutions. The final composite image 3110 can include the integratedview of the 2D image of the patient and the 3D bite model. The finalcomposite image 3110 and the final 3D bite model 3120 can be showcasedto the patient.

FIG. 16B depicts a method 1600B for virtually representing a treatmentoutcome using automated detection of facial and dental referenceobjects, according to some embodiments. Some or all of the operations inthe method 1600B may overlap with method operations discussed in thecontext of FIG. 16A. At an operation 1640, an initial position of thepatient teeth may be identified using a 3D representation of thepatient's teeth. The initial position may include the patient's teeth atan initial or at an intermediate stage of an orthodontic treatment plan.

At an operation 1642, an estimated interim final orthodontic position ofthe patient's teeth may be determined using a virtual representation ofone or more force systems applied to the patient's teeth. In variousimplementations, application of virtual representations of forces and/ortorques used as part of the orthodontic treatment plan may be modeledagainst the 3D representation of the patient's teeth.

At an operation 1644, one or more images of the patient's face may begathered. These images may be gathered from a camera or other imagecapture device as discussed further herein.

At an operation 1646, one or more facial reference objects on the imageof the patient's face may be identified. The one or more facialreference objects may correspond to a physical or anatomical featureproviding a first reference position to the patient's face. In someimplementations, a geometrical analysis of the image of the patient'sface may be performed, where desired RED proportions and/or anteriortooth widths are determined. The patient's “face type” may bedetermined.

At an operation 1648, one or more dental reference objects on the first3D representation of the patient's teeth may be identified. The dentalreference objects may correspond to physical or anatomical featureproviding a second reference position to the patient's dentition.

At an operation 1650, a relationship between the one or more facialreference objects and the one or more dental reference objects may beidentified. The relationship may involve measurement of distancesbetween the facial reference object(s) and the dental referenceobject(s). In some implementations, the relationship may involvealignment of the facial reference object(s) and the dental referenceobject(s).

At an operation 1652, the estimated interim final orthodontic positionof the patient's teeth in the first 3D representation may be modifiedbased on the relationship between the one or more facial referenceobjects and the one or more dental reference objects. In someimplementations, the modification may only occur if the relationshipmeets or is not greater than a specified threshold. As an example, themodification may only occur if a distance between the facial referenceobject(s) and the dental reference object(s) meets or does not exceed aspecified distance threshold.

At an operation 1654, instructions to integrate a modified estimatedinterim final orthodontic position based on the relationship between theone or more facial reference objects and the one or more dentalreference objects may be provided. In some implementations, the facialreference objects may be used as the basis of a target for the dentalreference objects. Examples include setting a dental midline thatmatches a facial midline; setting an incisal edge position that matchesa location of the inferior boarder of the upper lip at the facialmidline; setting an incisal edge position that matches a location of thesuperior boarder of the upper lip at the facial midline; and setting agingival zenith of a central incisor that aligns with the inferiorboarder of the upper lip at the facial midline. At an operation 1658,instructions to design and/or manufacture an orthodontic appliance usingthe modified estimated interim final orthodontic position are provided.

Matching a Bite Model with a 2D Image

FIG. 32 depicts a composite image 3200 of a 2D image of the face of aperson 3210 and a 2D image of a 3D bite model 3220. In the 2D image ofthe patient, the patient can have an open mouth. The opening of themouth 3230 is demarcated by edge of the lips 3214. The teeth 3260 andthe gingiva 3250 of the 3D bite model 3220 can be shown in the openingof the mouth 3230.

The 3D bite model 3220 can be a 3D digital visualization of the teeth,and it can be used to visualize the shape and position of the teeth in acurrent state or in a future state wherein the future state can be theshape and position of the teeth during or after undergoing a treatmentplan to reposition the teeth of the patient. The treatment plan caninclude orthodontic measures for correcting malocclusions and otherorthodontic issues with the patient's teeth, for example correcting anoverbite, an underbite, tooth rotation, or tooth tipping. In someembodiments, the treatment plan can include restorative measuresincluding but not limited to installing a crown, a veneer, or bridge.

The composite image 3200 can be used to provide facial context to thevisualization of the 3D bite model. In some embodiments, the compositeimage 3200 provides facial contextual visualization of the 3D model ofthe shape and position of the patient's natural teeth. In someembodiments, the composite image can provide facial visualization of the3D model of the shape and position of the teeth undergoing the treatmentplan. Such contextual information aids in the evaluation of the dentaltreatment plan and in determining the final position of the patient'steeth, for example when used with the treatment planning processes andsystems described herein. Contextual information also aids in thepatient's understanding of the treatment plan and in making an informedtreatment decision.

A composite image of a 2D image of the face of a person and a 2D imageof a 3D bite model (3D bite model) can create an unnatural look. Thisunnatural look may be described by the uncanny valley effect, which is ahypothesis that describes the adverse human emotional response to humanreplicas. Human replicas that appear nearly, but not quite like humanbeings elicit feelings of eeriness and revulsion. In contrast, humanreplicas that are distinctly different than humans do not elicit thisnegative emotional response. Composite images that include 3D bitemodels can seem almost, but not entirely, human when viewed in thecontext of a natural human face. Therefore, some composite images elicitfeelings of unease in the dental professional or in the patient. In someembodiments, an unnatural look can be caused by aesthetic differencesbetween the natural teeth and the 3D bite model, including but notlimited to, 3D teeth features, blur effect, teeth color, gingiva color,intensity of whiteness, intensity of red color, intensity of greencolor, and intensity of blue color. In order to aid in reducing theunpleasant reactions, the composite can be made to have a more seamlessintegration of the 3D bite model into the 2D image of the patient. Asdescribed below, systems and methods may be implemented to aid inmatching the 3D bite model with the 2D image of a patient. Variousfeatures of the 3D bite model can be controlled to aid this matching.Some embodiments of matching a 3D bite model with a facial image may useneural networks or machine learning algorithms, as shown and describedwith respect to FIGS. 36A and 36B.

FIG. 33 illustrates a controller 3300 that may be a digital userinterface for adjusting features of the 3D bite model 3220. Thecontroller 3300 may be part of the system 400 shown and described abovewith reference to FIG. 4. The controller 3300 may include toggleswitches and scaling bars that aid in adjusting features of the 3D bitemodel 3220. The controller 3300 can be used to control one or more of aplurality of features of the 3D bite model image. One of a plurality ofcontrollable features can be to adjust the 3D tooth effect, and the 3Dtooth effect can be turned on or otherwise adjusted via, for example, aswitch, such as the 3D teeth feature toggle switch 3310. One of aplurality of controllable features can be to create a blur effect, andthe blur effect can be turned on or otherwise adjusted via a switch,such as the blur model toggle switch 3311. One of a plurality ofcontrollable features can be used to adjust teeth color, and the teethcolor can be adjusted by or otherwise adjusted via a switch, such as theadjust teeth color toggle switch 3312. One of a plurality ofcontrollable features can be used to adjust teeth color options(indicated by the pick color bar 3313) using the press-button located inbar 3312 a dental professional may select a location on the 2D image foruse in coloring the 3D bite model. The tool color may also be adjustedusing preselected color options, which may be standard dental colors.For example, the color options may include an A1 color 3319, a B1 color3320, a C1 color 3321, a D2 color 3322, and an automatic color 3323chosen, for example, based on the colors of the teeth of the 2D image ofthe patient. One of a plurality of controllable features can be toadjust the intensity of whiteness, and the intensity of whiteness can beadjusted or otherwise selected by moving the slider 3324 along the scale3324 of the whiteness adjuster 3314.

One of a plurality of controllable features can also include adjustmentsand selection of the red, green, and blue color balance of the teeth ofthe 3D bite model. The controller 3300 includes a red adjuster 3325, agreen adjuster 3326, and a blue adjuster 3327 in respective colorselectors 3315, 3316, 3317. The adjusters and selectors may change thecolor intensity or balance of one both of the teeth and gingiva. Asshown in FIG. 33, the color adjusters and selectors adjust the colorbalance of the gingiva.

A quality metric, such as an integration quality metric, mayquantitatively describe the difference or degree of agreement between anintegrated composite image and a 2D image of a patient. In someembodiments, the quality metric is defined by a scale that correspondsto the degree of agreement or difference between the integratedcomposite image or the 3D bite model of the integrated composite imageand the 2D image of the patient or a portion of the 2D image of apatient, such as the image of the teeth and gingiva of the patient. Thescale of the quality metric may be a closed or open ended range. Forexample, in an open ended range the metric may start at 0, indicating aperfect match between the composite image and the image of the patient,and increase without limit based on the degree of agreement, withincreasingly high numbers indicating increasingly poor match oragreement between the images. In a closed ended range 0 may indicate aperfect match while 1, 10, or 100, may indicate a complete mismatchbetween the images or a mismatch above a predetermined threshold betweenthe images.

FIG. 34 illustrates a 3D bite model in the composite view with variousdegrees of match between the respective composite view 3410, 3420, 3430and a reference 2D image of the patient. The matching may be performedwith the controller, for example as described above with reference toFIG. 33 or according to the method 3500 described below with referenceto FIG. 35, or another method, such as that shown and described withreference to FIGS. 36A and 36B. The composite image 3410 shows anexample of a composite image with a high degree of mismatch and lack ofagreement with a reference image. In such an image, the mouth opening3411, delineated by the inner edges of the lips 3412 bear little to noresemblance with the mouth, teeth, and gingiva of an image of thepatient. The quality metric of such a composite image may be high, suchas greater than a threshold value.

The composite image 3420 shows an example of a composite image with amoderate degree of mismatch and lack of agreement with a referenceimage. In such an image, the mouth opening 3411, delineated by the inneredges of the lips 3412 includes a colored 3D bite model 3425 of thepatient within the mouth opening 3411 with basic color and shading addedsuch that the teeth 3424 and gingiva 3423 are apparent, but significantand noticeable differences may still exist between the 2D image of thepatient. The quality metric of such a composite image may be high, suchas greater than a threshold value, but lower than that of the compositeimage 3410.

The composite image 3430 shows an example of a composite image with alow degree of mismatch and high degree of agreement with a referenceimage. In such an image, the mouth opening 3411, delineated by the inneredges of the lips 3412 includes a colored 3D bite model 3435 of thepatient within the mouth opening 3411 with accurate color and shadingadded such that the teeth 3434 and gingiva 3433 include few, if any,noticeable differences between the 3D bite model 3435 and the image ofthe patient. The quality metric of such a composite image may be low,such as zero, or less than a threshold value of acceptable agreement.

FIGS. 35 and 36 illustrate a method of matching the 3D bite model withthe 2D image of a patient. The process for obtaining the 3D bite modelin integration view 3500 can be based on a comparative analysis wherebythere is a minimization of the differences between a first image and asecond image. The minimization process may reduce the quality metric toat least a threshold value of acceptance. FIG. 36 shows a reference 2Dimage 3641 of the patient, including their natural teeth 3642 andnatural gingiva 3643, along with three images of a composite image,including a composite image 3610 at an initial stage, a composite image120 at an intermediate stage of processing, and a composite image 3630at a final stage of processing.

The process for obtaining the 3D bite model 3500 may include multiplesteps and may be iterative. At block 3510 the default settings for therendering parameters are set. Such default parameters may serve as theinitial conditions of the matching process. The rendering parameters canbe derived from the parameters of natural teeth 3642 and natural gums3643 in the 2D image 3641. Subsequently, as depicted in block 3520, thevalue of the integration quality metric can be determined by comparingthe composite image to the 2D image of the patient. The comparisonbetween the composite image 3610 and the 2D image 3641 of the patientmay include a pixel by pixel comparison of the differences between thepixels in the composite image and the corresponding pixels in the samelocation in the 2D image of the patient.

In some embodiments, the quality metric, determined based on thedifferences in the pixels, may be the Mean Absolute Error (MAE) of thepixels compared between the two images. In some embodiments, the qualitymetric, determined based on the differences in the pixels, may be theMean Squared Error (MSE) of the pixels compared between the two images.The difference in the pixels may include the difference between one ormore of the red, green, blue, and luminance values of each pixel. Insome embodiments, the quality metric may be based on a peaksignal-to-noise ratio, such as the PSNR-HVS-M or PSNR-HA methods.

In some embodiments, other methods for calculating the quality metricmay be used. For example, databases of scored images may be used in thequality metric, for example, a metric comparison using the TID2008,TID2013, LIVE, Toyama, IVC, CSIQ, and others may be used. In someembodiments, perceptual visual quality metrics such as FSIMs, SSF,PSNR-HAc, SR-SIM, BMMF, and others may be used. The metric comparisonused in calculating the quality metric may include the use of Spearman'scorrelation, the Kendall correlation, or others.

In block 3530 a step of minimization may be performed where theparameters, such as the rendering parameters, of the 3D bite model areadjusted to aid in reducing the quality metric to below a thresholdvalue. The parameters may include teeth color, gingival color, materialproperties and light properties. Color parameters may include colortemperature, red, green, and blue color intensities, saturation,luminance of the teeth and gingiva. Material parameters may includesurface texture modifications such as surface textures and surfacereflectivity. Light parameters may include shading (such as thevariation of shading on the teeth between the front teeth, which areexposed more directly to light, and the back teeth, which are lessexposed to outside light due to the light blockage by the lips andcheeks), light source location, and type, such as a point source ornon-point source of light. The process alters these parameters in such away as to minimize the quality index. In some embodiments, as part ofthe minimization process, a Monte Carlo method may be implementedwherein various parameters are modified and an estimation of theireffect on the quality index is determined. The parameters of the 3D bitemodel may then be modified based on the results of the Monte Carloanalysis.

At block 3540 the new value of the integration quality metric may becalculated based on the image of the 3D bite model generated using therevised parameters. The interim composite image 3620 shows the interimintegrated 3D bite model 3641 with the interim parameters applied. Atblock 3550 the quality metric is compared to a threshold value. If thequality metric is acceptable based on the threshold value, then theprocess proceeds to block 3560. The quality metric may be acceptable ifthe value of the quality metric is below the threshold value. If thequality metric is not acceptable based on the threshold value, thenanother step of minimization and calculation of the quality metric maybe performed as in blocks 3530 and 3540. As a result of this process,the 3D bite model may more closely match that of the image of thepatient. In some embodiments, the 3D bite model may have parameters thatare similar to or the same as the parameters of the 2D image 3641. Afinal composite image 3630 may be generated based on applying theparameters determined earlier in the process to the 3D bite model 3631such that the 3D bite model 3631 is matched to the colors of the naturalteeth 3642 of the 2D image 3641 of the patient. Such a process mayresult in a seamless and natural integration of the 3D bite model intothe image of the patient.

In some embodiments, neural networks, such as generative adversarialnetworks or conditional generative adversarial networks may be used tointegrate a 3D model of teeth in a final position with a facial image ofa patient and match the colors, tones, shading, and other aspects of the3D model with a facial photo. FIG. 36A depicts an embodiment of a method4800 of integrating a 3D model of a patient's teeth in a clinical finalposition with a facial image of a patient. At block 4802, the neuralnetwork is trained using facial images. In some embodiments, the facialimages may include images of people's faces having a social smiles. Insome embodiments, the facial images may include facial images ofpatient's teeth before orthodontic treatment. During training, patient'steeth and their contours may be identified. For example, each tooth maybe identified by type (e.g., upper left central incisor, lower rightcanine). Other aspects and features of the image may also be identifiedduring training, such as the location and color of the gingiva, thecolor of the teeth, the relative brightness of the surfaces within themouth, and others.

Referring to FIGS. 36A and 36B, after training, the neural networkreceives inputs at block 4804 for use in generating a realisticrendering of the patient's teeth in a clinical final position. In someembodiments, the inputs may include one or more of an image of arendering of a 3D model of the patient's teeth in a clinical finalposition or a 3D rendered model of the patients teeth in the clinicalfinal position 4808, the clinical final position determined, forexample, according to an orthodontic treatment plan, a blurred initialimage of the patient's teeth 4810, and a color coded image 4812 of the3D model of the patient's teeth in the clinical final position.

The image of a rendering of a 3D model of the patient's teeth in aclinical final position or the 3D rendered model of the patients teethin the clinical final position 4808 may be determined based on theclinical orthodontic treatment plan for moving the patient's teeth fromthe initial position towards the final position, as described above. Theimage or rendering 4808 may be generated based on the imagingperspectives determined as described with respect to FIG. 13E. Forexample, one or more of the imaging distance, the focal length of theimaging system, and the size of the patient's teeth in the initialfacial image may be used to generate the image or rendering.

The blurred image of the patient's teeth 4810 may be generated using oneor more blur algorithms, such as a Gaussian blur algorithm. The Gaussianblur preferably has a high radius, for example, a radius of at least 5,10, 20, 40, or 50 pixels. In some embodiments, the blur is sufficientlygreat that the tooth structure is no longer readily apparent to a humanobserver.

The coded model of the patient's teeth 4812 may be a red-green-blue(RGB) color coded image of a model of the patients teeth, with eachcolor channel corresponding to a different quality or feature of themodel. For example, the green color channel, which may be an 8-bit colorchannel indicates the brightness of the blurred image 4810 on a scale of0 to 255 as, for example, overlaid on the 3D model.

The red color channel may be used to differential each tooth and thegingiva from each other. In such an embodiment, the gingiva may have ared channel value of 1, the left upper central incisor may have a redvalue of 2, the right lower canine may have a red channel of 3, theportions of the model that are not teeth or gingiva might have a redchannel value of 0, and so on, so that the red channel value of eachpixel identifies the dental anatomy associated with the pixel.

The blue color channel may be used to identify the angle of the teethand/or gingiva with respect to the facial plane. For example, at eachpixel location the angle normal of the surface of the dental structure,is determined and a value between 0-255 (for 8-bit color channels) isassigned to the pixel. Such information allows the neural network to,for example, model light reflectivity from the dental surfaces.

At block 4806 the neural network uses the inputs and its training torender an image realistic image of the patient's teeth in a finalposition. This photo realistic image is then integrated into the mouthopening of the facial image and an alpha channel blurring is applied.

The method 4800 provides a significantly more realistic integration ofthe 3D model with the facial image than previous methods. This pushesthe realism beyond the uncanny valley.

Tooth Mass Reduction for Restorative Objects

Patients can display symptoms of malformed teeth or injured teeth.Malformed or injured teeth may be chipped, broken, worn down throughgrinding or other means, or simply malformed.

A restorative object can be used to treat one or more injured ormalformed teeth. A restorative object, such as an artificial tooth orartificial part of the tooth, restores or corrects the shape of aninjured or malformed tooth. In some embodiments, a restorative objectmay be a crown, which is a tooth-shaped cap that is placed over a toothto restore its shape and size and improve its appearance. In someembodiments, the restorative object may be a veneer, which is a thinshell of material that covers the front surface of a tooth. Therestorative object can have a predetermined shape and size. In order toaid the fitting of the restorative object over the tooth, the tooth mayundergo tooth mass reduction where a portion of the tooth is removed toprovide a mounting surface to receive the restorative object.

Different processes for applying restorative objects may use differingamounts of tooth reduction. For example, applying a veneer to a toothmay result in less tooth mass reduction than applying a crown. Differingamounts of tooth mass reduction may also be seen between differentapplications of similar restorative objects. For example, applying aveneer to a tooth in a first position may involve removing more toothmass than applying a veneer to the same tooth, but in a differentposition. Evaluating tooth mass reduction for restorative objects indifferent positions may aid in reducing or minimizing the amount oftooth mass loss due to the application of restorative objects.

FIG. 37 depicts an illustration of teeth with restorative objectsapplied thereon. In some embodiments, such as those shown in FIG. 37,the root 3710 and 3720 of the tooth extending to the neck 3711 and 3721may remain intact. As shown in FIG. 37, a prosthetic crown 3714 may beapplied to a reduced tooth 3712 or a veneer 3714 may be applied to areduced tooth 3722.

FIG. 38 depicts how the amount to tooth mass loss is determined. To aidin fitting the restorative object to a tooth, the tooth may undergotooth mass (or volume) reduction where a portion of the tooth is removedand the remaining tooth is a tooth with a reduced mass or volume. Asshown in FIG. 38 the crown 3832 of the tooth is chipped or otherwisebroken. The broken line represents the reduced portion of the crown 3840after tooth mass reduction. In this embodiment, the remaining reducedportion 3840 serves as the mount for the restorative object, such as aprosthetic crown. The mass or volume 3846 removed from the tooth inpreparing the tooth for the restorative object is determined, forexample, by subtracting the volume of the reduced crown 3840 from thevolume of the natural crown 3842.

In some embodiments, the removed tooth mass is healthy tooth mass but isremoved in order to create space for the addition of the restorativeobject. It may be beneficial to the patient if the amount of removedtooth can be reduced or minimized. The amount of tooth mass that isremoved can be determined with a 3D bite model during the treatmentplanning process. The position and shape of the restorative object canbe visualized with the integration of the restorative object on the 3Dbite model.

FIG. 39 illustrates a 3D bite model 3900 that depicts teeth, gingiva3940, a restorative object on the right canine 3910, a restorativeobject on the right lateral incisor 3920, and a restorative object onthe right central incisor 3930. The visualization of the restorativeobject in the 3D bite model may be useful in determining the shape orposition of the restorative object on the patient's natural teeth. Theability to determine the shape of the restorative object before dentalwork, such as orthodontic corrections, may allow for tooth massreduction analysis, where the amount of tooth loss is determined forvarious restorative object shapes on teeth in one or more finalorthodontic positions. The restorative object shape that uses the leasttooth mass loss may be beneficial to the patient.

Dental professionals and technicians may modify one or both of therestorative object's placement and the orthodontic final teeth positionsto minimize or reduce tooth mass reduction. The 3D bite model can beused to determine a movement a tooth of a patient from a first positionand orientation to a second position and orientation as describedherein. Computer-based treatment planning can be used to facilitate thedesign of each tooth trajectory in the treatment plan, as describe abovewith reference to FIGS. 1-4. During the treatment planning process theposition and shape of the restorative object may be modified to matchthe treatment goals. As shown in FIG. 40, the process for obtaining the3D bite model may begin with determining an initial tooth position 4010,such as an initial position for one or more teeth of the patient. Insome embodiments, the initial positions of the teeth are determined asdescribe above with reference to FIGS. 1-4. At block 4020, the interimfinal orthodontic positions of the teeth are determined, for example asdescribed above with reference to FIGS. 1-4.

At block 4030 the interim restorative object position is determined. Theprocess for determining an interim restorative object position 4030 maybegin by adjusting the orthodontic position to an interim orthodonticposition at block 4041 and then determining at new interim restorativeposition at block 4032. The tooth mass loss of the new interimorthodontic position is compared to the tooth mass loss of one or moreof the previous interim orthodontic and restorative positions at block4033 wherein the new interim restorative position is evaluated. At block4033 the final orthodontic position and the restorative object positionis determined. In some embodiments, the final orthodontic position andthe restorative object position is determined by selecting the finalteeth positions that minimize tooth mass or volume loss among theevaluated positions. In some embodiments, the positions may bedetermined based on minimaxing certain restorative procedures, such asminimizing the number of crowns. In some embodiments, less invasiveprocedures, such as the use of veneers, are given priority over moreinvasive restorative procedures, such as crowns and root canals or wholetooth extraction and prosthetics. This process allows for the evaluationof a new restorative position at each stage of the orthodontic treatmentplan. The various interim restorative positions provide multiple optionsduring the orthodontic treatment plan of when to apply the restorativeobject to the patient. The amount of tooth mass reduction can bedifferent at each interim orthodontic position, thus providing moreoptions.

Restorative Object Position on 3D Models

The restorative object is a part of the final 3D bite model and can bemodified to reach the treatment goals. One treatment goal may be theaesthetic and functional placement of the teeth. For example, in someembodiments, the final or target position of the teeth in a treatmentplan may include positioning and shaping the teeth such that theirshapes and positions conform to Recurring Esthetic Dental (RED)proportion. The dimensions of the boxes of the proportion widget aredetermined by the Recurring Esthetic Dental (RED) proportion. Under REDproportions, the successive width proportion when viewed from the facialaspect should remain constant from the midline toward the posterior forthe six anterior teeth between and including the canines. This propertyoffers great flexibility to match tooth properties with facialproportions. RED proportion may aid in determining the final shape andsize of one or more restorative objects and the final or target positionof the patients teeth. The 3D bite model can be assessed by determiningthe various dimensions of the patient's teeth, for example the widget4260 illustrates the demarcation of the shape the teeth. FIG. 42illustrates the various dimensions and positions of the teeth that maybe determined. The dimensions of the medial incisor 4211, the lateralincisor 4221, and the canine 4231 may be measured. These dimensions mayinclude the width of the medial incisor 4210, the width of the lateralincisor 4220, and the width of the canine 4230. Also, the intercaninewidth (ICW) may be determined based on the distance between the rightand the left canine 4231. The height of each tooth 4250 can also bedetermined from the 3D bite model. The initial dimensions of the teethare determined based on the initial 3D bite model built from the scan ofthe patient's teeth and using the midline plane and occlusal plane.

FIG. 43 illustrates a 3D bite model of the upper jaw from both theanterior position on the left and from the occlusal place, on the right.

FIG. 44 illustrates the modified shape and position of the restorativeobjects on a 3D bite model 4400. The final shape and position of therestorative object for each tooth of a plurality of teeth may be shapedaccording to the RED proportion as shown by the bounding boxes 4460. Thewidth of the central incisor 4410, lateral incisor 4430, and the canine4440 are indicated by the bounding boxes. Similarly, the height of thecentral incisor 4450, lateral incisor 4452, and the canine 4454 areindicated by the bounding boxes.

In some embodiments, neural networks, such as generative adversarialnetworks or conditional generative adversarial networks may be used tointegrate a 3D model of teeth in a final position with a facial image ofa patient and match the colors, tones, shading, specular highlights andother aspects of the 3D model with a facial photo, for example, asdiscussed above, in particular with respect to FIGS. 36A and 36B.

FIG. 45 depicts an embodiment of a method 4500 of integrating a 3D modelof a patient's teeth in a clinical final position with a facial image ofa patient. At block 4502, the neural network is trained using facialimages. In some embodiments, the facial images may include images ofpeople's faces having a social smile. In some embodiments, the fiscalimages may include facial images of patient's teeth before orthodontictreatment. During training, patient's teeth and their contours may beidentified. For example, each tooth may be identified by type (e.g.,upper left central incisor, lower right canine). Other aspects andfeatures of the image may also be identified during training, such asthe location and color of the gingiva, the color of the teeth, therelative brightness of the surfaces within the mouth, and others.

Referring to FIGS. 45 and 46, after training, the neural networkreceives inputs at block 4504 for use in generating a realisticrendering of the patient's teeth in a clinical final position. In someembodiments, the inputs may include one or more of an image of arendering of a 3D model of the patient's teeth in a clinical finalposition or a 3D rendered model of the patients teeth in the clinicalfinal position 4508, the clinical final position determined, forexample, according to an orthodontic treatment plan. The inputs may alsoinclude a blurred initial image of the patient's teeth 4510 or a colorcoded image 4512 of the 3D model of the patient's teeth in the clinicalfinal position or both.

The image of a rendering of a 3D model of the patient's teeth in aclinical final position or the 3D rendered model of the patients teethin the clinical final position 4508 may be determined based on theclinical orthodontic treatment plan for moving the patient's teeth fromthe initial position towards the final position, as described above. Theimage or rendering 4508 may be generated based on the imagingperspectives determined as described with respect to FIG. 13E. Forexample, one or more of the imaging distance, the focal length of theimaging system, and the size of the patient's teeth in the initialfacial image may be used to generate the image or rendering.

The blurred image of the patient's teeth 4510 may be generated using oneor more blur algorithms, such as a Gaussian blur algorithm. The Gaussianblur preferably has a high radius, for example, a radius of at least 5,10, 20, 40, or 50 pixels. In some embodiments, the blur is sufficientlygreat that the tooth structure is no longer readily apparent to a humanobserver.

In some embodiments, the blurred image of a is a parametric blurredimage or a parametric blurring model, for example, as described in U.S.application Ser. No. 16/041,613, filed Jul. 20, 2018, the contents ofwhich are incorporated by reference herein.

The coded model of the patient's teeth 4512 may be a multi-channelrepresentation of the patient's teeth. The coded model may include aplurality of channels for each location in the model. Each channel mayinclude information, such as data, regarding one or more characteristicsof each respective location in the model. For example, channels mayencode the normal angles for each location on each tooth or for eachlocation of the surfaces depicted in the coded model. A channel mayencode the luminosity with respect to the normal for each locationdepicted in the model. A channel may include coded data with respect todepth of each location in the model within the oral cavity.

In some embodiments, the coded model of the patient's teeth 4512 may bea red-green-blue (RGB) color coded image of a model of the patientsteeth, with each color channel corresponding to a different quality orfeature of the model. For example, the green color channel, which may bean 8-bit color channel indicates the brightness of the blurred image4510 on a scale of 0 to 255 as, for example, overlaid on the 3D model.

The red color channel may be used to differentiate each tooth and thegingiva from each other. In such an embodiment, the gingiva may have ared channel value of 1, the left upper central incisor may have a redvalue of 2, the right lower canine may have a red channel of 3, theportions of the model that are not teeth or gingiva might have a redchannel value of 0, and so on, so that the red channel value of eachpixel identifies the dental anatomy associated with the pixel.

The blue color channel may be used to identify the angle of the teethand/or gingiva with respect to the facial plane. For example, at eachpixel location the angle normal of the surface of the dental structureis determined and a value between 0-255 (for 8-bit color channels) isassigned to the pixel. Such information allows the neural network to,for example, model light reflectivity from the dental surfaces.

In some embodiments, the neural network may receive, as an input, apreprocessed image of the patient's mouth. The preprocessed image may bebased on the patient photo, a mask of the patient's mouth, a coded imageof normals of the intra-oral surfaces, a depth map, which may be a codedimage includes data related to the depth of the patient's oral structurewithin the oral cavity, and a segmentation mask, which may be a maskthat segments the rendered 3D model. For example, the segmentation maskmay define masks that segment the upper gingiva, lower gingiva, upperarch, and lower arch from each other. In some embodiments, thesegmentation mask may segment each individual tooth from each other. Thepreprocessed image may be formed based on these inputs and then itself,used as an input to the neural network.

In some embodiments, the inputs to the neural network may have aresolution with a ratio of 2 to 1, for example, 512×256 pixels, 256×128pixels, 1024×512 pixels, and etc.

At block 4506 the neural network uses the inputs and its training torender a realistic image of the patient's teeth in a final position.This photo realistic image is then integrated into the mouth opening ofthe facial image and an alpha channel blurring is applied to the image.In some embodiments, to the lip boundary. In some embodiments, theblurring is applied only to the lip boundary.

At block 4514 the color of the rendered teeth are adjusted. FIG. 47shows a method 4700 of adjusting the color of the rendered teeth. Themethod 4700 may start at block 4702, wherein the teeth are masked. Themasking process may use the color coded model 4512 as the mask for theteeth. In some embodiments, a mask of the teeth is generated todifferentiate the portions of the model or rendered image that are teethfrom the portions of the model or rendered image that are not teeth.

The method may start at block 4704 instead of block 4702. At block 4704,the color space of the rendered image of the teeth is converted to theL*a*b* color space. The L*a*b* color space, also referred to as theCIELAB color space, uses three numerical values, L*, a*, and b* todefine the color for each pixel in a rendered image. The L* is theluminance or lightness channel, and may range from 0-100 with 0representing the darkest luminance and 100 representing the white point,or brightest value, for example. a* is the green-red color channel andmay range from −128 to +127 with all green and no red being representedby −128 and all red and no green being represented by +127. Similarly,b* is the blue-yellow color channel and may range from −128 to +127 withall blue and no yellow being represented by −128 and all yellow and noblue being represented by +127.

After block 4702 and 4704, at block 4706 the teeth are brightened. Insome embodiments, the teeth are brightened by increasing the value ofthe L* channel of the area of the rendered teeth defined by the maskcreated at block 4702. In some embodiments, the value of the L* channelfor pixels of the area defined by the mask is increased. In someembodiments, the value of the L* is shifted towards the brighter end ofthe L* scale.

At block 4708 the teeth are whitened, for example, by adjusting the b*channel more towards the blue end of the scale. In some embodiments, theteeth are whitened by the decreasing value of the b* channel of the areaof the rendered teeth defined by the mask created at block 4702. In someembodiments, the value of the b* channel for pixels of the area definedby the mask is decreased.

A single tooth whitening treatment reduces the b* channel by an averageof 3 levels and increases the brightness in the L* channel by an averageof 1 level (or 1% nominally). At blocks 4706 and 4708 the amount ofbrightness and whiteness can be added in a natural manner by ensuringthat the teeth do not trend from yellow to blue.

Referring back to FIG. 45, at block 4516 specular highlights are addedto the rendered tooth model or the final combined photo. Specularhighlights are a significant contributing factor in the realism ofrendered image, such as a composite image described herein. Specularhighlights occur on shiny surfaces where the angle between the lightsource and the surface normal is similar to or the same as the anglebetween the viewer and the surface normal. Often, the location of thelight sources in the image are unknown so the light source cannot bemodeled during the 3D rendering of the tooth model in order to recreatespecular highlights on the teeth that match the light on the patient'sface. FIG. 48 depicts a method 4900 of creating realistic specularhighlights without knowing the position of the light source of the 2Dphoto of the patient's face.

At block 4902, the teeth are masked. The masking process may use thecolor coded model 4512 as the mask for the teeth. In some embodiments, amask of the teeth is generated to differentiate the portions of themodel or rendered image that are teeth from the portions of the model orrendered image that are not teeth.

At block 4904, the bright locations in the area defined by the maskcreated at block 4702 are located. The bright locations can beidentified base on, for example, the luminance of each pixel in theimage. In some embodiments, the luminance value of each pixel iscompared to a threshold value, of the luminesce value indicates that thepixel is brighter than the threshold value, such as 97, then a speculahighlight is identified. In some embodiments, when the luminance of apixel is greater than a threshold luminance value, then a specularhighlight is located. In still other embodiments, a specular highlightmay be located based on the luminance value of the pixel being equal toa maximum luminance value, such as 100.

At block 4906 the normal at each pixel location in the teeth isidentified. For example, as discussed above, the tooth surface normalmay be determined from the blue color channel used to identify the angleof the teeth and/or gingiva with respect to the facial plane. In someembodiments, the color coding described with reference to FIG. 36B mayinclude the direction of the normal to the surface of the tooth at eachlocation or pixel on the surface of the tooth.

At block 4908 the normal at each specular location is identified basedon the location of the specular highlights identified at block 4904 andthe normals identified at block 4906. In some embodiments, the set ofnormals may be a single average normal of the normals at the specularhighlight locations or a plurality of the most common normalsidentified, such as the 5 or 10 most common normals. In someembodiments, the average normal is identified and the set of normalsincludes the average normal plus or minus 1 degree, 2 degrees, 3degrees, 4, degrees, or 5 degrees. In some embodiments, the distributionof normals at which specular highlights are located may be multi nodal.In such an embodiments, the normal at each node is identified. Such anembodiment may include embodiments in which multiple light sources areused to illuminate the face of a patient with acquiring the facial imageof the patient. In some embodiments, the set of normals includes some orall of the normals at which the specular highlights occur.

At block 4910 the specular highlights are applied to the patient's teethin the final position. The specular highlights may be applied to eitherthe 3D model of the patient's teeth or the 2D rendering of the patient'steeth in the final position.

In some embodiments, to apply the specular highlights an average normalis computed from the set of specular highlight normal. In someembodiments, outlier rejection, such as the RANSAC method, may be usedto when determining the average normal. Then, the cosine similaritybetween the average normal and every tooth normal in the final toothimage or 3D model is determined. The location and magnitude of thespecular highlights may then be determined using the Blinn-Phong shadingmodel by computing a ΔL*=B*D{circumflex over ( )}S, where S is therelative shininess of the teeth (e.g., 100), D is the cosine similarityat the location, and B is the brightness of the specular highlights(e.g. 50). The computed ΔL* (the increase in brightness based on thecomputation of the contribution of the specular highlights to thebrightness of the teeth at each location) can be added to the L* channelof the L*a*b* color image of the teeth to create simulated specularhighlights in the proper location for the lighting in the image. Such amethod allows the simulation of specular highlights on an image withoutknowing the location of the light sources in the image.

In some embodiments, the mean and covariance of the set of normals isdetermined. Then, a multivariate normal distribution is used with thespecular highlight mean (M) and covariance (C). The degree of specularhighlights at each location is then be determined based on ΔL*=Bpdf(D|M,C), where B is the brightness of the highlights, M is the mean,C is the covariance, pdf is a probability density function based on Mand C, and, as above, ΔL* is the increase in brightness based on thecomputation of the contribution of the specular highlights to thebrightness of the teeth at each location. The determined ΔL* for eachpixel in the image or location on the model of the teeth can be added tothe L* channel of an L*a*b* color image of the teeth to create simulatedspecular highlights in the proper location for the lighting in theimage.

The method 4500 provides a significantly more realistic integration ofthe 3D model with the facial image than previous methods. This pushesthe realism beyond the uncanny valley.

While preferred embodiments of the present invention have been shown anddescribed herein, it will be obvious to those skilled in the art thatsuch embodiments are provided by way of example only. Numerousvariations, changes, and substitutions will now occur to those skilledin the art without departing from the invention. It should be understoodthat various alternatives to the embodiments of the invention describedherein may be employed in practicing the invention. It is intended thatthe following claims define the scope of the invention and that methodsand structures within the scope of these claims and their equivalents becovered thereby.

What is claimed is:
 1. A method of orthodontically treating a patient'steeth comprising: receiving facial image of the patient that depicts thepatient's teeth; receiving a 3D model of the patient's teeth;determining color palette of the depiction of the patient's teeth;coding 3D model of the patient's teeth based on attributes of the 3Dmodel; providing the 3D model, the color palette, and the coded 3D modelto a neural network; processing the 3D model, the color palette, and thecoded 3D model by the neural network to generate a processed image ofthe patient's teeth; simulating specular highlights on the processedimage of the patient's teeth; and inserting the processed image of thepatient's teeth into a mouth opening of the facial image.
 2. The methodof claim 1, comprising: forming spline at the edge of the inner lips todefine the mouth opening of the facial image.
 3. The method of claim 1,comprising: training the neural network using facial images of peoplethat depict their teeth.
 4. The method of claim 1, comprising: blurringthe processed image of the patient's teeth.
 5. The method of claim 4,wherein the blurring occurs after inserting the processed image of thepatient's teeth into the mouth opening of the facial image.
 6. Themethod of claim 5, wherein the blurring is alpha channel blurring. 7.The method of claim 1, wherein generating the color palette comprises:blurring the depiction of the patient's teeth from the facial image. 8.The method of claim 7, wherein the blurring is a Gaussian blur.
 9. Themethod of claim 1, wherein the facial image is a 2D facial image. 10.The method of claim 1, wherein coding the 3D model comprises coding acolor channel of a plurality of pixels of a 2D rendering of the 3D modelwith attributes of the 3D model or the facial image.
 11. The method ofclaim 10, wherein the attributes are one or more of the brightness ofthe patient's teeth at each pixel location, the angle of the surface ofthe 3D model with respect to the facial plane at each pixel location,and the dental structure type of the 3D model at each pixel location.12. The method of claim 11, wherein the brightness of the patient'steeth location at each pixel location is determined based on thebrightness of a blurred depiction of the patient's teeth from the facialimage.
 13. The method of claim 11, wherein the dental structure is oneor more of an identity of each tooth or the gingiva in the 3D model ateach pixel location.
 14. The method of claim 1, wherein the processedimage of the patient's teeth is a 2D image.
 15. The method of claim 1,wherein simulating specular highlights on the processed image of thepatient's teeth includes: identifying bright locations on the patent'steeth in the facial image; determining the surface for a plurality oflocations on the patient's teeth in the facial image; identifying a setof normals at the bright locations; and applying simulated specularhighlights in the processed image based on the set of normals.
 16. Themethod of claim 15, wherein the set of normals are an average normal ofthe normals at the specular highlight locations.
 17. The method of claim15, wherein the set of normals a plurality of the most common normalsidentified.
 18. The method of claim 15, wherein the set of normalsincludes an average normal plus or minus 1 degree, 2 degrees, 3 degrees,4, degrees, or 5 degrees.
 19. The method of claim 15, wherein the set ofnormals includes each node of a multimodal distribution of normals. 20.The method of claim 15, wherein the set of normals includes the normalsat which the specular highlights occur.
 21. The method of claim 15,wherein applying the simulated specular highlights in the processedimage based on the set of normals includes: determining an averagenormal from the set of normals at the bright locations. identifyingsecond normals at each location of the teeth in the processed image;determining the cosine similarity between the average normal and thesecond normals at each location; determining a difference in brightnessat each location of the teeth in the processed image based on the cosinesimilarity at each respective location; and applying the difference inbrightness to a brightness channel pixels of the processed image at eachrespective location.
 22. The method of claim 15, wherein determining theaverage normal from the set of normals at the bright locations includesusing outlier rejection.
 23. The method of claim 15, further comprising:determining the difference in brightness at each location based on abrightness, a specular highlight brightness, a relative shininess of theteeth, and the cosine similarity at each location.
 24. The method ofclaim 23, wherein determining the difference in brightness at eachlocation based on a brightness a specular highlight brightness, arelative shininess of the teeth, and the cosine similarity at eachlocation includes using a Blinn-Phong shading model.
 25. The method ofclaim 15, wherein the processed image uses the L*a*b* color space. 26.The method of claim 1, further comprising: brightening the patient'steeth in the processed image.
 27. The method of claim 26, whereinbrightening the patient's teeth in the processed image comprises:masking the patient's teeth by created a tooth mask; increasing theluminance channel of the image in the tooth mask; and increasing theblue-yellow color channel of the image in the tooth mask.
 28. The methodof claim 26, wherein increasing the luminance channel of the image inthe tooth mask includes increasing the value of a L* channel by
 1. 29.The method of claim 26, wherein increasing the blue-yellow color channelof the image in the tooth mask includes increasing the value of a b*channel by
 3. 30. The method of claim 26, further comprising: convertingthe processed image to L*a*b* color space.